November 6, 2012

Launchpad blog
lp-blog
Launchpad blog
The information sharing feature is complete

Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

The Disclosure features

Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.

There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing.   We did not consider Sharing, or any of the required features complete until we fixed all the bugs.

Disclosure facts

  • Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
  • Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
  • 149 related bugs were identified in Launchpad.
  • Work started in June 2011 by the Purple squad.
  • Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
  • About 45,000 lines code were added to support the features.
  • About 15% of the lines were for missing JavaScript test coverage.
  • More that 700 bugs were fixed in total.
  • About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
  • About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.

Lessons learned

  • Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
  • Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
    • Privacy was added on top of bugs, and it failed to scale to 100′s of bugs.
    • Privacy was added on top of branches, and it failed to scale to 1000′s of branches.
    • Filtering private items in code, or in database joins is not fast enough to work with 100,000′s of items.
  • Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
  • Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
  • Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.

August 28, 2012
Information sharing is now in beta for everyone

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

./share-projects-with-team.py my-team project1 project2

July 13, 2012
Launchpad does not have private projects…yet.

Nothing breaks my heart quite like a request to make a project private–make it invisible to everyone except to the people the project trusts. I am utterly crushed when someone who works for Canonical or on Launchpad asks for one. I have been planning this feature for more than two years, and the Purple squad has been working on it for 13 months. I blog about this, I send emails about this, I present reports on this, but the people who most need private projects don’t know what the Purple squad is doing. I think the problem here is that Launchpad squads no longer use Launchpad to plan and execute work. There is no place for any interested party to see what the goals of Disclosure is and gauge how we are progressing.

I present my first draft of a report that states the simple goals of that the Disclosure feature wants to achieve . The report provides some summaries of the work that allows anyone to see what the Purple squad is doing, recently done, and will do next. There is also some analysis that provides insight into the amount of work remaining. This report complements the Purple squad’s kanban board. While kanban is excellent for tracking branches of code and technical tasks, the level of detail is unsuitable for non-Launchpad developers. The kanban board is also only accessible to a small number of people. I want a report that anyone interested in private projects or managing the disclosure of private information can see and understand. Mostly, I want everyone to see that the Purple squad is delivering valuable features and know when we will be done.

I based the report on the intended reporting UI for Launchpad series and milestones. I really miss using series and milestones to plan releases. For every milestone, I wrote our goals in the summary, and targeted bugs to the milestone. Though we abandoned the analytics because of performance concerns, I could reliably judge  the contributors’ velocity, and see when I needed to retarget work to another milestone because the remaining effort exceeded the milestone’s work capacity. Though I didn’t provide a burn down chart of the work, I could sort the milestone to see the colour change. I could confidently see and predict 3 months of work.

This report replaces the canvas-based chart I planned for series and milestones with a YUI 3 chart. The listing of bugs are split into categories so that I can focus on scheduling or provide Diogo with a list of bugs that need exploratory testing. Though this report thinks it is talking to Launchpad, it is actually using JSON for the 500+ bugs that I pulled using a trivial Launchpad API script. Since the data is cheap to retrieve, I can load the chart multiple times, each looking a different set of bug tags so that I can see specific themes of work.

The report shows that there is more than 60 days of work to complete the features needed by private projects. The Orange squad will work on private projects while the Purple squad finishes the prerequisites.

 

June 25, 2012
Creating teams on demand

teams

We want the project maintainer to be the default party that the project shares private information with. The problem at the moment is that Launchpad does not know how to set a team as the project maintainer during setup. Improper project setup is the root cause of most cases where information is disclosed to the wrong people. We need to improve project registration and setup to ensure users can ensure private information is managed properly. This issue is complicated by a very old issue, it is not possible to register a team at the moment you discover you need one. Launchpad must let me register a new team that will maintain my project when I first setup my project.

The Purple Squad discussed what we can do to simplify team registration and perform the registration in any page that allows you to set a team. We discovered several areas where we can make improvements.

  • Do not ask for non-essential information like contact address.
  • We can simplify the team membership policy language.
  • We can fix the confusion about membership renewal.
  • Launchpad can pre-fill the form with sensible defaults when the team will be used in a role.

Ian put together a demonstration to prove we could extend the person picker to also permit you to register a team.

When you want to set the project maintainer to a new team, Launchpad will ask you to confirm its suggestions for the Launchpad Id, display name, and membership policy. You can change the values, but most of the time you will choose to continue, and Launchpad will register the team and place it in the role.

 

April 18, 2012
Setting up commercial projects quickly

Setting up a commercial project in Launchpad has gotten easier. You can now quickly register a proprietary project and enable private bugs. You can create private teams and private personal package archives, AKA private PPA or P3A without the assistance of a Launchpad admin.

When you select the Other/Proprietary license while registering a project, or changing the project’s details,

it is given a complimentary 30-day commercial subscription.

The delay between the moment when a commercial project was registered and when the commercial subscription was purchased and then applied to the project caused a lot of confusion. During this delay, proprietary data could be disclosed. We chose to award the project with a short term commercial subscription which enabled the project to be properly configured while the 12-month commercial subscription was being purchased and applied to the project.

Any project with a commercial subscription can enable

Default private bugs
Once enabled by configuring the project’s bug tracker, all new reported bugs are private. You can choose to make the report public.
Default private bugs
Default private branches
You can request a Launchpad admin to configure private branches for your teams. (You will be able to do this yourself in the near future when projects gain proprietary branches.)

As the maintainer of a project with a commercial subscription, you can register

Private teams
When you register a team, you can choose to set the team visibility to private. The team’s members and data is hidden from non-members.
Private mailing lists
When you create a mailing list for a private team, the archive is also private. Only team members may see the messages in the archive.
Private PPAs
When you create a PPA for your public team, you may choose to make it private; private teams can only have private PPAs. You can subscribe users to your archive so that they may install packages without revealing all your team’s members and data to the subscriber.

A secondary benefit of this change is that you can now try Launchpad’s commercial features before purchasing a 12-month commercial subscription. The features will be disabled at the end of 30-days. Your test data will remain private to ensure your data is not disclosed.

Any open source project may also have a commercial subscription to enable commercial features. You can purchase a commercial subscription at the Canonical store. Commercial subscriptions cost US$250/year/project + applicable V.A.T.

 

(Photo by Fred Dawson on flickr, creative commons license)

April 13, 2012
An introduction to our new sharing feature

Launchpad can now show you all the people that your project is sharing private bugs and branches with. This new sharing feature is a few weeks away from being in beta, but the UI is informative, so we’re enabling this feature for members of the Launchpad Beta Testers team now. If you’d like to join, click on the ‘join’ link on the team page.

What you’ll see

Project maintainers and drivers can see all the users that are subscribed to private bugs and branches. The listing might be surprising, maybe even daunting. You may see people who no longer contribute to the project, or people you do not know at all. The listing of users and teams illustrates why we are creating a new way of sharing project information without managing bug and branch subscriptions.

If you’re a member of (or once you’re a member of, if we want people to join) the Launchpad Beta Testers team, you can find the Sharing link on the front page of your project. I cannot see who your project is sharing with, nor can you see who my projects are sharing with, but I will use the Launchpad project as an example to explain what the Launchpad team is seeing.

The Launchpad project

The Launchpad project is sharing private bugs and branches with 250 users and teams. This is the first time Launchpad has ever provided this information. It was impossible to audit a project to ensure confidential information is not disclosed to untrusted parties. I still do not know how many private bugs and branches the Launchpad project has, nor do I even know how many of these are shared with me. Maybe Launchpad will provide this information in the future.

Former developers still have access

I see about 30 former Launchpad and Canonical developers still have access to private bugs and branches. I do not think we should be sharing this information with them. I’m pretty sure they do not want to notified about these bugs and branches either. I suspect Launchpad is wasting bandwidth by sending emails to dead addresses.

Unknown users

I see about 100 users that I do not know. I believe they reported bugs that were marked private. Some may have been subscribed by users who were already subscribed to the bug. I can investigate the users and see the specific bug and branches that are shared with them.

The majority

The majority of users and teams that the Launchpad project is sharing with are members of either the Launchpad team or the Canonical team. I am not interested in investigating these people. I do not want to be managing their individual bug and branch subscriptions to ensure they have access to the information that they need to do their jobs. Soon I won’t have to think about this issue, nor will I see them listed on this page.

Next steps — sharing ‘All information’

In a few weeks I will share the Launchpad project’s private information with both the Launchpad team and the Canonical team. It takes seconds to do, and about 130 rows of listed users will be replaced with just two rows stating that ‘All information’ is shared with the Launchpad and Canonical teams. I will then stop sharing private information with all the former Launchpad and Canonical employees.

Looking into access via bug and branch subscriptions

Then I will investigate the users who have exceptional access via bug and branch subscriptions. I may stop sharing information with half of them because either they do not need to know about it, or the information should be public.

Bugs and private bugs

I could start investigating which bugs are shared with users now, but I happen to know that there are 29 bugs that the Launchpad team cannot see because they are not subscribed to the private bug. There are hundreds of private bugs in Launchpad that cannot be fixed because the people who can fix them were never subscribed. This will be moot once all private information in the Launchpad project is shared with the Launchpad team.

Unsubscribing users from bugs

Launchpad does not currently let me unsubscribe users from bugs. When project maintainers discover confidential information is disclosed to untrusted users, they ask the Launchpad Admins to unsubscribe the user. There are not enough hours in the day to for the Admins to do this. Just as Launchpad will let me share all information with a team or user, I will also be able to stop sharing.

 

(Image by loop_oh on flickr, creative commons license)

January 26, 2012
Social private teams

The title may sound like a contradiction, but I assure you that it is not. You can now use private teams in social contexts in Launchpad. Private teams can collaborate with public projects and other teams if they choose to reveal the existence of the private team.

  1. Private teams can me members of other teams.
  2. Other teams can be members of private teams.
  3. Private teams can subscribe to bugs, blueprints, branches, and merge proposals
  4. Private teams  can be in public project roles,  such as maintainers, drivers, and bug supervisors.
  5. You can change the branch owner to be a private team.
  6. Private team personal branches (+junk) are always private.

When a member places the private team in a social situation, a the member is asked to confirm that it is okay to reveal the team’s identifying information. This information is the team’s Launchpad Id used in URLs, the displayname, icon, and logo. A user can visit the private team’s Launchpad page and will only see that information. The rest of the page is not shared. Anonymous users cannot see a private team’s page because that user is not being social; logged in users can see the private team’s page

Private team page seen by a non-member

Launchpad did not permit these interactions previously because it was not clear who should know about the team. Someone has to know. If Launchpad permitted private teams to subscribe to bugs or be members of teams without anyone knowing of them, they would be unaccountable. Private teams could spy on organisation, or learn about security vulnerabilities to exploit. Launchpad will not ever permit such asocial behaviour. The resolution for social interactions was to permit other parties to know enough of the private team to make an informed decision. For example, when I choose to make a bug private, I want to know who was already seen the bug through a subscription. I may choose to unsubscribe the private team if I do not trust them.

Private teams may invite exclusive teams to be members. Exclusive teams (moderated or restricted teams) review all members so they are accountable. If a team admin trusts the admins of another team, and that team is accountable, Launchpad permits the other team to be a member. This is actually a rule that applied to all exclusive teams. private teams are always exclusive (restricted membership policy). The only nuance with private teams is when it is a member of another team; the super team may know the members of the private sub team because the super team has the right to audit all its members so that it too can be accountable.

November 8, 2011
Removing a project from a shared bug report

Launchpad will soon permit you to say your project is not affected by a bug shared with another project — you can delete the spurious bug task. This action can be done from the bug’s web page, and using Launchpad API.

You can remove a project from a bug if you are the project maintainer, bug supervisor, or are the person who added the project to the bug. This action can only be performed when the bug affects more than one project — you cannot delete an entire bug. This feature permits you to undo mistakes.

Launchpad beta testers will see the remove action next to the affected project name in the affects table.

Delete spurious bugs

The delete() method was added to the bug_task entry in the Launchpad API. There is a example API script, delete_bugtasks.py, that can remove a project from many bugs. There is also a split action to create a separate bug just for the specified project to track separate conversations in bug comments.

Usage: delete_bugtasks.py [options] bug_id:project_name [bug_id:project_name]

Options:
  -h, --help            show this help message and exit
  -v, --verbose
  -q, --quiet
  -f, --force           Delete or split bug tasks that belong to a public bug
  -d, --delete          Delete a spurious project bug task from a bug
  -s, --split           Split a project bug task from an existing bug so that
                        the issue can be tracked separately
  -w WEBSITE, --website=WEBSITE
                        The URI of Launchpad web site.
                        Default: https://api.launchpad.net;
                        Alternates: https://api.staging.launchpad.net,

https://api.qastating.launchpad.net

Previously, you could not remove spurious bug reports about your project. Many were cause by poor bug target management; you could not move a bug between projects and distributions. You can now move bugs between projects and distributions, but thousands of bugs still wrongly claim to affect a project or distribution. This causes clutter on bug pages and searches, and it causes Launchpad performance problems.

This change is a part of a super-feature called Disclosure. To ensure that confidential data is not accidentally disclosed, Launchpad will only permit private bugs to affect a single project. Soon, you may need to remove a project from a bug before marking the bug private.

May 19, 2011
Nikki and the Robots

Nikki and the RobotsNikki and the Robots is the first game from Berlin based games studio Joyride Labs. It’s a retro-styled platformer with beautiful colours and an open source licence, with bugs tracked in Launchpad.

I asked Iwan & Sönke from Joyride Labs about the game.

Matthew: Nikki and the Robots is LGPL and Creative Commons licensed. What made you choose open source licences?

Iwan & Sönke: This one is easy! Our love of open source software and free art made us do it! Also we expect that being open will get us additional attention and love.

Our work and how we license it is an experiment too, so actually we ‘hope’ rather than expect. :)

Matthew: How are you planning to distribute the game once it’s ready?

Iwan & Sönke: We are working on the first part of Nikki and the Robots and will start in-dev-sales (pre-sale) once it is ready.

Game, editor and user levels will be free as in freedom. A part-proprietary organic story mode with enhanced levels and additional in-game-art will be available for pay.

Matthew: Can you tell me a bit more about the technology behind the game and the decisions you made? I see, for example, you’re using Haskell.

Iwan & Sönke: Haskell is just my (Sönke’s) favourite language. I have the impression that I can do coding in Haskell much faster than in other languages and I produce less bugs. Besides Haskell we use Qt and OpenGL as the graphics backend and Chipmunk as the physics engine. By the way: we completely rely on free software for compilation on all platforms (so, no VisualStudio or XCode, just gcc, cmake, mingw32, etc.).

Matthew: How are you finding Launchpad as a bug tracker?

Iwan & Sönke: It’s quite usable. Forum and wiki might be useful features to consider for the platform. (We use a free wiki system and will probably set up an own forum soon).

Matthew: Thanks to Iwan and Sönke. You can download an alpha of Nikki and the Robots from the Joyride Labs website.

May 12, 2011
Beer as in beer

JolieBulle logoThere’s quite a bit of overlap between home beer brewing and hacking. Both usually involve experimentation, sharing and a love of what you’re doing.

It’s not surprising, then, that there’s more than one open source project aimed at helping home brewers to create the beer they want. A few home brewing projects are hosted on Launchpad, including JolieBulle (that’s French for “pretty bubble”).

Home brew ... cider in this casePierre Tavares started the project last year to support his own brewing. The result is an application that helps at every stage of the process. When you’re ready to get going, it helps formulate the recipe and allows for the sharing of recipes using the common BeerXML standard. It helps calculate what brewers call the beer’s profile (its bitterness, colour, how much alcohol it has), includes an ingredients database and has tools that help during the brewing process itself.

I emailed Pierre to ask about the development of JolieBulle. Here’s what he said:

From a technical point of view, JolieBulle is developped in pyQt and integrates well in both KDE and Gnome desktops.

I chose Launchpad mainly for the openness of the platform, and the great tools to manage code, bugs and blueprints. I’m pretty new to DVCSes but Bazaar seems fine, and I have no problem using it. I don’t use the translation tool, as I prefer Qt Linguist.

JolieBulle isn’t yet packaged for any distros and Pierre hopes to attract contributors who can help with that.

Photo by Sizbut. Licence CC BY 2.0.

May 9, 2011
Doing it for the Luz

Luz is a new project to Launchpad. It creates impressive visual effects that can react to music or be driven by a person using a MIDI controller or a gamepad.

It has been created by Ian McIntosh, part of a Portland, Oregon, artists collective who produce light and projection shows.

Here’s what the Luz page on the Light Troupe site has to say:

With one click, any movement or effect can dance to the beat, react to audio, or be driven directly by human input from any number of any device: Gamepads & Joysticks, MIDI knobs & sliders, MIDI Pianos & Drums, WiiMotes, Wacom Tablets, and any app that can send OpenSoundControl.

Ian has provided a handy series of YouTube tutorials, to get you started. If you want to try it out, here’s the first of those tutorials:

April 26, 2011
Gufw in Launchpad

GUFW's logoIf you’re an Ubuntu user, it’s likely that you’ve used UFW — the Uncomplicated Firewall — and maybe also its graphical front-end, Gufw.

I spoke to Marcos Costales from Gufw to ask about the project and their use of Launchpad.

Matthew: What prompted you to start the project?

Marcos: In 2008 I was translating and testing, but I wanted to contribute more. When Canonical released ufw, I read some reviews that a firewall using the shell was not an uncomplicated firewall; reviews did not see that ufw was perfect for servers and a base with great ideas for something else, I saw a clear opportunity for the development of an application like Gufw.

Matthew: How many of you are working on it?

Marcos: After the first week following release, Vadim joined the project and we were able to generate a lot of expectation. It was amazing to see how Gufw grew exponentially thanks to the contribution of the community. An important point was Cedrick’s mockup for setting a good interface.

From here I would like to thank Vadim, Emilio, Raúl, Planella, Rubén, Rogério, Cedrick and Devid, Gufw is what it is thanks to them.

We are currently working on an application that really simplifies the task of setting up a firewall: very minimalistic, easy to use and understand.

Matthew: How much connection is there between the Gufw and UFW communities?

Marcos: From Gufw we pay attention to the next ufw versions and we request changes we need. We want to thank Jamie for his effort and interest in all of our requests.

There is also an awesome work of documentation taking place in the Ubuntu Wiki.

Matthew: Does Launchpad help the two projects work together?

Marcos CostalesMarcos: Yes, we can follow the blueprints of the next ufw version and review which changes could affect Gufw. We can easily reassign ufw and gufw bugs…

Matthew: Why did you choose Launchpad?

Marcos: Gufw started in Google Code. After one month we moved to Launchpad, and it certainly was one of our best decisions because it offers us all necessary services and an amazing integration between them.

Matthew: What’s your favourite Launchpad feature?

Marcos: Translations. I am a translator in external projects to Launchpad, and I must say that the usability and simplicity of Launchpad is unique.

Matthew: What would you most like to see added to Launchpad?

Marcos: Maybe a small space for a custom website. We must use an external hosting for the main project website.

Matthew: What do you think is Launchpad’s biggest weakness?

Marcos: At first Launchpad seems overly complex and big (compared for example with Google Code). One idea would be to have two views according to the needs/experience of the project: the current one and an easier to use one.

Matthew: Are you looking for people to contribute?

Marcos: Of course we are. There are always pending tasks for anyone who wants to help and everyone is welcome.

Matthew: Thanks Marcos!

April 20, 2011
Tomdroid: Tomboy notes for Android

I like to use Tomboy for just about any time I need to make a note. Shopping lists, a GTD collection bucket, notes from a phone call: Tomboy’s ideal.

Actually, up until recently I didn’t use Tomboy for shopping lists all that often. Making the list was fine but getting it to the supermarket usually meant printing it out, or something else that wasn’t quite as convenient as I’d like.

Baskets only

Then I installed Tomdroid.

Tomdroid does one thing and it does it well: it synchronises your Tomboy notes from elsewhere and lets you read them on your Android phone. You can import your notes from an SD card or, more usefully, synchronise with a TomboyWeb provider such as Ubuntu One.

Now I can tap out my shopping list on a real keyboard and carry it with me without a second thought. And, of course, all those other notes that make my life run smoothly are there with me wherever I am.

Tomdroid uses Launchpad for bug tracking, code hosting, blueprints and questions/answers.

You can download the latest version from the Android Market.

Photo by Tristram Biggs. Licence: CC BY-ND 2.0.

April 19, 2011

Curtis Hovey
sinzui
Sinzui » Sinzui
Creating a signed release file with setup.py

I recently created https://launchpad.net/lp-release-manager-tools for some videos I am making about how to use Launchpad. One aspect of creating a release is to upload the source tarball and a detached gpg signature verifying it. This is somewhat ironic, since lp-release-manager-tools exists to automate repetitive tasks that release managers do in Launchpad. I really do not like creating the signature. I cannot remember how to do it. I need to read the instruction on the form to upload the tarball each time. So I added a feature to my example project that any project hat uses python distutils can copy to make the signature with the source tarball.

I subclassed the sdist command and added an extra step to create the detached signature of the tarball. I then register the new command as signed_dist. This is the content of my setup.py:


#!/usr/bin/python
import subprocess

from distutils.core import setup
from distutils.command.sdist import sdist

class SignedSDistCommand(sdist):
    """Sign the source archive with a detached signature."""

    description = "Sign the source archive after it is generated."

    def run(self):
        sdist.run(self)
        gpg_args = [
            'gpg', '--armor', '--sign', '--detach-sig', self.archive_files[0]]
        gpg = subprocess.Popen(
            gpg_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        gpg.communicate()

setup(
    name="lp-release-manager-tools",
    description="Launchpad release manager API scripts.",
    version="0.0.2",
    license="MIT",
    maintainer="Curtis C. Hovey",
    maintainer_email="sinzui.is@verizon.net",
    scripts=['close_released_bugs.py'],
    cmdclass={
        'signed_sdist': SignedSDistCommand,
        }
    )

October 28, 2010

Launchpad blog
lp-blog
Launchpad blog
Nautilus Terminal

Nautilus Terminal in action

If you’re a Gnome user and have watched with envy as your KDE4-using friends effortlessly open a terminal directly in their file-browser, you may be interested in Nautilus Terminal.

Fabien Loison is behind Nautilus Terminal. I asked him a little about the project.

Matthew: What were you doing when you realised that life would be easier if you had a terminal in Nautilus?

Fabien: I was programming and I had a lot terminals open on different folders. I realized I was losing a lot of time to find the one I wanted, and then I remembered that Midnight Commander has an interesting feature: it permits you to enter commands in the current folder. I searched on the web and saw that (KDE 4’s file manager) Dolphin offers this kind of functionality, but nothing about Nautilus… So I decided to do it myself: I started programming Nautilus Terminal.

Matthew: And are you happy with the result?

Fabien: Although Nautilus Terminal is not as well integrated with Nautilus I would like (due to limitations of its extension system), I think I have solved my problem. :)

Matthew: What sort of reaction have you had?

Fabien: Most reactions were positive: since the day of the first release I have received many emails and also some blogs have written about Nautilus Terminal (WebUpd8, OMG Ubuntu,…). It seems that many people wanted this feature in Nautilus.

Matthew: So what made you choose Launchpad?

Fabien: I had already been using Launchpad for other projects for several months, and I like it (especially its integration with Bazaar), so I used it one more time. :)

Matthew: What has been the most useful part of Launchpad?

Fabien: The most useful part of Launchpad for this project has been the bug tracker, because there were a lot of problems in the first versions.

Matthew: And, similarly, where would you like to see Launchpad improve?

Fabien: That is a difficult question… Maybe having a small wiki for every projects (for the documentation).

Matthew: Finally, are you looking for contributions from other people?

Fabien: Yes, especially for the translations, because I can’t do it myself for all languages (I can translate in French only). So thanks to all translators (and all the people who have helped with code, bug reports,…). :)

Matthew: Thanks Fabien!

Visit Nautilus Terminal in Launchpad.

May 5, 2010
The Economist and Launchpad

Economist logoThe online team at The Economist recently set up a Launchpad project, using a commercial subscription. I asked Mark Theunissen, from The Economist Group, about their plans.

Mark: We’re migrating the existing Economist.com stack from Coldfusion/Oracle to a LAMP stack running Drupal. At present, we’re about half way through — if you visit a blogs page, channel page, or comments page they will be served from Drupal, but the home page and actual articles are still served from Coldfusion. There’s a migration and syncronisation process happening in the background between Oracle and MySQL.

Matthew: Is much of your web infrastructure based on open source software? If so, what?

Mark: Our new stack sure is! :) We run almost all open source, in fact I can’t think of anything that isn’t.

  • Redhat Linux servers throughout (not Ubuntu, unfortunately).
  • MySQL enterprise database.
  • PHP 5.
  • Varnish HTTP accelerator.
  • Drupal content management system. Actually, a distribution called Pressflow.
  • Memcached for caching.
  • BCFG2 for configuration management.
  • The Grinder for load testing.

Matthew: Do you customise much of that?

Mark: We do, yes. We’ve sponsored or contributed patches that have mostly been for Drupal but also made their way into Varnish & BCFG2. We use Pressflow, and our changes go there first and often get back ported into core Drupal. Our policy is to open source as much as humanly possible!

Matthew: And, of course, I’d love to know what made The Economist choose Launchpad.

Mark: We chose Launchpad for its usability, mostly the workflow around reviewing code (merge proposals). It provides excellent tools for managing distributed teams, and we are a very large distributed team, with three locations where development is occurring on either side of the Atlantic.

The integration with Bazaar is great, and we are going to consider moving our bug tracker to Launchpad too at some time in the future.

Matthew: Thanks Mark!

April 26, 2010
Mopedix

Thomas Pietrowski is an 18 year old high school student in Solingen, Germany, whose mopedix project aims to create a control system for mopeds and other vehicles.

Matthew: Tell us more about mopedix

Thomas: My software system is split into two different parts.

First of all there is the control system, which is using an Arduino to control relays and H-bridges that also can dim light. This project is still under construction and soon I’ll be adding functions such as locking via a RFID reader and activating an alarm using an accelerometer.

But the important part of the project is the interaction between the control system and the computer. The client can be used for setting your values, e.g. when you want the moped to turn on its lights according to the intensity of the ambient light. In general it sends commands via serial connection and this gives the project a wide range of ways to communicate with the Arduino used to control the moped’s lights.

At the moment I am working with the Arduino Duemilanove, which has a built-in USB-to-serial adaptor. But there is also the possibility of communicating using a cable, e.g. RS232 or simply two wires, one for incoming data and one for outgoing data, or wireless via Bluetooth [50-100m], radio frequency [2-10m], Xbee[>90m-1.6km (1 mile)] and many more. Using Bluetooth you can also change your settings using a Nokia mobile phone running PyS60 or other devices that can send commands via the serial connection to the control system.

But to be honest, the control system has not been tested on the vehicle, as I still need to transport it from my grandparents in Poland to Germany. However, I’ve tested that it works in theory.

Matthew: What prompted you to start writing software that interacts with your moped?

Thomas: I decided to start the project in the end of 2009 when I looked on the web to see if there was something similar already available. I found some discussions in forums and elsewhere but didn’t come across anything that really did what I wanted.

The most important thing that interested me was how much such a control system would cost, because prices on the market are in the most cases not very realistic. Imagine a radio frequency controlled locking system kit for your car. Such a kit can cost around 100 Euros. Now think about making such a locking system on your own using Bluetooth and your mobile phone.

Here is a short calculation what you would need:

  • a bluetooth-serial module (14 Euros from China)
  • an Arduino board (20 Euros)
  • a 6v relay that can handle voltages of 12v (2 Euros)
  • and some other typical things like a few transistors, resistors, cables, and a solder clip (5-7 Euros).

That makes a total of 41 to 43 Euros and some hours programming your Arduino and a little Python script for your Nokia S60 mobile phone.

So, can you see it can be cost effective to offer such a system.

Another thing is that I had an Arduino Duemilanove and had been developing some applications in Python for my personal use, so I had the most important things that were needed to start my project: the resources and the knowledge.

Because of the great community, lots of Arduino users, a detailed instructions for programming the Arduino and my own Python experience, I decided to give it a go.

Matthew: What sort of interface does a moped have that allows you to hook it into a computer?

Thomas: You can use mopedix in general on any vehicle you want. I aimed it at mopeds because, as an 18 year old student, I only have a moped and access to its schematics.

By reading its schematics I noticed that the heart of my moped was actually the ignition lock. The control system that I am working on is nothing more than a digitally controlled ignition lock”, which will replace the old one and provide an interface for the client application.

The system will be powered on my moped with 6 volts and newer mopeds that have 12 volts will need at least a 9V rectifier for use with the Arduino Duemilanove or a 5V rectifier for the Arduino Mini. An Arduino Mini will be great for the control system because I believe that there will be also vehicles that will have room available than in my moped.

It should be possible to use the same software and control system on a car. So, for the future I should think about a project name that isn’t specific to a category of vehicle.

Matthew: Is there any other software, proprietary or open, that does a similar job?

Thomas: Not that I know. I found, via Google, a patent that describes a control system for bicycles, but I am sure that there is no other software, neither proprietary nor open, that can be compared with mine.

In addition to that I would really like to learn from people who are familiar in working on free software and earning money for their work to show me how to earn money on my own for buying additional modules, like a bluetooth-serial-module, to improve my software and provide the end-user a wider range using the control system.

Matthew: Why did you choose Launchpad?

Thomas: I chose Launchpad because I worked on packages that are hosted in my PPAs for the Canola project (that page is a little outdated) and I have been helping to test unstable Ubuntu packages since Lucid Alpha 3.

So I was already familiar with Launchpad but the main reason for choosing Launchpad was that it gives the possibility to other people translating my client application into their languages from English.

Moreover it gives the possibility for the end-user to get in contact with the project using the “Answers” tab on the project page or report errors by using the “Bugs” tab. As well as that the user is able to follow the future of the project by the “Blueprints” tab. But I haven’t used this feature yet because I didn’t think that there were people who would be interested in my project.

The only reason I don’t use Bazaar is that I’ve only previously worked with Subversion. I hope to take some time to learn Bazaar, in the future, switch over from Subversion.

Matthew: Thanks for telling us about mopedix!

February 17, 2010
Sikuli — scripting your use of GUIs

Sikuli logo
The Sikuli project recently switched to using Launchpad. I asked Tsung-Hsiang Chang to tell me more about the project.

Matthew: Your website says, “Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots).” That sounds like it’s a general purpose screen-scraping solution. Is that right?

Tsung-Hsiang: Not exactly. Sikuli is not about extracting data from screen.

The current release of Sikuli is called Sikuli Script, which focuses on only automation using screenshots of GUI widgets.

We have another project called Sikuli Search, which queries a search engine using screenshots instead of keywords.

Although Sikuli Script is supposed to be able to “search” buttons or text on the screen, it isn’t good at scraping or analyzing information from screenshots yet.

Matthew: In your YouTube demo video, you show how you can write a Sikuli script that will open an app by name. Then, you can take a screen shot of one of the icons in that app and have Sikuli click it as part of the script. You even show how you just have to take a screen shot of the option you want from a drop-down list and Sikuli will select that option. That’s really cool but you say that Sikuli will even tolerate small changes in the icons you ask it to click. How does that work?

Tsung-Hsiang: Matching between a target image and the screen image is done by computing the normalized cross-correlation between the two images.

This is a standard technique in computer vision for finding patterns when variations are known to be small. This technique works incredibly well for matching desktop GUI patterns.

Matthew: Where do you think Sikuli will be most useful?

Tsung-Hsiang: Whenever the internal API of an application is not exposed. Lots of people have created their scripts to play the applications that were used to be very difficult to be automated, such as facebook (flash) games and testing android systems.

Matthew: Particularly on Mac OS X and Linux-based systems, where does Sikuli become a better option than standard shell scripting?

Tsung-Hsiang: If a command line interface is available, Sikuli may be not a good choice for a shell scripting guru. But sometimes you just can’t find command line tools or don’t want to learn complicated commands and parameters. In fact, the core of Sikuli is just a Jython library, so it can be mixed with other Python scripts or command line tools easily. Therefore, Sikuli can be an additional handy tool for command line gurus.

Besides, the primary goal of Sikuli is to help ordinary users who know nothing about command line tools and shell scripting to automate their tasks. We hope everyone can enjoy using computer efficiently.

Matthew: What’s next for Sikuli?

Tsung-Hsiang:Better, faster, more accurate.

We have a long list of planned improvements for Sikuli. Among the top of list are:

  1. Social programming: the ability to share scripts and search scripts by visual patterns. For example, when a user takes a screen shot of a recycle bin, the user can search a database for all the other scripts written by other users that involve the image of a recycle bin.
  2. Event-driven programming: the ability to register event handlers to handle visual events. For example, a user can define a function to pop up a warning message to handle the visual event that involves the appearance of the “low battery image”.
  3. Face detection: the ability to find faces on the screen.
  4. Recorder: the ability to record a sequence of clicking and typing operations and generate a visual script automatically.
  5. Tutorial converter: the ability to convert an existing step-by-step instruction with screenshots into executable scripts.

Matthew: Why did you choose Launchpad?

Tsung-Hsiang:We were using Trac before moving to Launchpad. Trac is more developer-oriented, just like Github.

But what we really want is a user-oriented project hosting site. We want a place to report and discuss bugs, ask and answer questions, and also download and track the development of Sikuli.

We compared Github and Launchpad, and at last chose Launchpad over Github because Launchpad has almost everything we need, except a wiki for writing documents. But we already had a wiki in our Trac system, so this was not a big problem.

Matthew: Do you have any requests for the Launchpad community?

Tsung-Hsiang: It would be great if we can write documents on Launchpad. A wiki or something like EtherPad would be fantastic.

Matthew: Thanks very much and good luck with Sikuli!

November 9, 2009
Ibid chat bot

Chat bots can be controversial. Apart from the fun to be had watching a friend carrying out an inadvertent Turing test, finding the right balance between useful information and annoyance seems to be one of the harder aspects of chatbotology.

I spoke to Jonathan Hitchcock, Michael Gorven and Stefano Rivera about their work on the Ibid chat bot.

Matthew: What are you doing with Ibid that’s different to other chat bots?

Ibid team: Like all bots, Ibid has two parts: the front end, and the back end, and we like to think that there is something to distinguish Ibid in both.

On the front end, we attempt to make the bot as intuitive as possible to interact with. All commands are natural language, and our philosophy is that the plugins should try to work out what the user actually wants to know and answer that, rather than forcing him/her to learn an esoteric set of syntax before he can use the bot. Queries are bubbled through plugins in order of priority so that the most relevant reply can be given.

Secondly, on the back end, Ibid is also divided into two parts: “sources” (i.e. transports, communication mediums – IRC, jabber, email, etc) and “plugins” (modules that tell the bot how to respond to various queries). The design aims to be clean and modular: all sources and plugins are discrete units that can be enabled or disabled without breaking any assumptions elsewhere in the code. Unlike other bots where non-IRC protocol support seems to be an afterthought, Ibid plugins deal with all protocols equally (hopefully without making them all equally bad).

This modularity helps both new and existing developers. It is easy to jump straight into a small part of the code without having to understand the entire system. It is also very easy to add new sources: we had a DC chat source up and running in under a day, and a prototype GSM SMS gateway in a weekend. In addition, hot-pluggability is an important goal of the project: you can add and remove plugins and sources on the fly, without having to restart the bot.

Matthew: Who do you see using Ibid?

Ibid team: As we’ve been developing the bot, we’ve started thinking in terms of three different personas: users, owners and developers.

Users are the end-users who happen to be in IM channels where an Ibid is running — they should just see the bot as a fun and useful tool. They’ll notice people using it to google things, store factoids, check the weather, do currency conversions, and so on, and start following suit. We’ve noticed that our bots very quickly become integral to channel communities, and we try to make sure the bot’s “personality” facilitates this: the natural language interface, and the characteristic responses that the bot gives aid this. Michael and Jonathan both have Ibid instances running internally at their companies, and the other employees find them very useful.

Owners are the people that run instances of the bot — they probably run the channels too, and they need a way to integrate other things with IM. Ibid aims to make that easy. We’ve already got plugins for checking RSS feeds, working out distances, fetching tweets, and so on — the plugin nature of the bot’s features mean that as soon as a new need arises, we can churn out some code to fulfill it.

This is where the third persona comes in: the developer. Ibid is designed to have a very low barrier to entry, and to make it very easy to write plugins, so it can be used as a framework for interfacing with almost anything. It already has plugins for interfacing with Bazaar, Trac and Buildbot, and a Launchpad plugin is in development. Because Ibid is so modular, stripping out all the funky stuff to build an IM interface to a single service is pretty simple, and Ibid hopes to be a good building block for such systems. Which brings us to your next question…

Matthew: Are you hoping to attract developers to Ibid?

Ibid team: See here.

That would be a yes. In the channels where we have Ibids running, users have contributed plugins for features they want, and in one case used it as a quick way to get an IRC game up and running. We’d like more of all of those.

We’ve done our best to make it as simple as possible to write Ibid plugins. There’s still some work to go (mostly documentation), but anyone who knows some basic Python should be able to hack a quick plugin together in a very short time.

We haven’t made an initial release yet, as some of the Internal APIs have some biggish problems that need to be resolved, but the bot is packaged and already works out of the box, and we’d love for people to start using it and contributing plugins.

Matthew: The project’s a fairly heavy user of Bazaar. What made you choose Bazaar over other VCSs?

Ibid team: It was almost accidental — we didn’t have much experience with DVCS, but we definitely wanted to use one rather than, say, subversion. We saw that the Ubuntu community was punting Bazaar, which is relatively sane (compared to, say Git), and ran with it. It turned out to be a good choice: it’s simple to learn and pretty powerful. It also has a very low barrier to entry, coming from more traditional VCSs, but you can ramp up to using the advanced features in fairly small, but logical steps (which makes it very similar to python, as a learning language, I suppose).

Matthew: What aspect of Launchpad has most helped your development of Ibid?

Ibid team: We originally started with Launchpad just to get quick bug-tracking integration, but soon discovered the other features on offer. After moving the source over to be wholly hosted on Launchpad, we started to use branches and merge-requests, and these have now become integral to our development methodology. Every feature and fix that we work on is developed in a separate branch and the merge reviewed by all three of us. Waiting for reviews can stagnate development a bit when all the developers are busy with their dayjobs, but we think the results are worth it — apart from the obvious benefits of catching bugs before they’re merged into trunk, the request-and-review process keeps us all involved and aware of what is happening in the project.

I imagine that as we grow, this benefit will lessen (since not everybody will review each merge), but it is still a good way of signalling to the rest of the team that changes are happening in an area, so that if they are interested, they can make their opinions heard before the changes get finalised.

As a whole, the tight Lauchpad-Bazaar integration has been a huge bonus, allowing us to develop from a few guys committing bits of code to trunk into a community with workflows, milestones, goals and a clear vision of where we’re headed. Having the VCS and the bug-tracker and the discussion boards and everything all in one place really builds cohesion in the project.

Matthew: What would you most like to see improved or otherwise changed in Launchpad?

Ibid team: We have been moving more and more of our project management into Launchpad (throwing out Trac, and our own Bazaar repository, and our own mailing lists, and so on), but there are still a few things that we have to host ourselves: repositories for non-Ubuntu packages (just Debian debs right now, but we’d like to include RPMs, etc, in the future), the build environments to create them (we have our own pbuilders presently), documentation, and our wiki.

There are free services that provide each of the above, such as the OpenSUSE build service, and the many “create your own wiki” services, but filling those gaps would mean our project could live entirely on Launchpad, and would bring along greater integration and easier flows.

Other neat things we could use would be support for Bazaar hooks, and permanently linking merge commits with the merge request — currently we include the merge request URL in the commit log.

Basically, tighter integration of everything would make project management much easier, and let us focus more on coding, which is always a win.

Matthew: Thanks Jonathan, Michael and Stefano!

September 11, 2009
Xibo: open source signage

Xibo is a free software signage system. I asked its lead developers, Alex Harrington and Dan Garner, about the project.

Matthew: So, does Xibo power the sorts of display we see in train stations and airports?

Alex and Dan: Yes, and in shops, post offices, schools, universities, colleges, churches or hotels. Anywhere there is a need to display information to the public or employees via a screen or projector. There’s over 150 Xibo installs worldwide that we know of, running 250 screens — however the Launchpad download stats suggest the actual install base is much larger.

Xibo allows organisations to build up collections of images, videos, RSS feeds, Flash, Embedded or External HTML and Microsoft Powerpoint presentations and combine them together into presentations which can then, in turn, be scheduled on to one or more of the displays attached to the Xibo system.

Matthew: We’re pretty used to seeing the Windows blue screen of death on public displays. In building Xibo, what have you done to ensure high uptime and to avoid rather public, embarrassing, error screens?

Alex and Dan: The current stable release of Xibo (1.0.3 at the time of writing) comprises a server application written in PHP and a client application written in C#. The client is written with resilience in mind, and is capable of operating over poor quality internet connections or running offline for periods of time.

The classic BSOD FAIL images from digital signs are often the underlying operating system crashing rather than the signage application itself. To that end, the client attempts to be as light as possible on system resources to keep the workload on the OS manageable – We have a production client running on a Via EPIA Fanless Mini-ITX system.

Looking forward, we have a new cross-platform (Linux/Win/Mac) client coming written in Python using the libavg engine which is guaranteed BSOD free! It uses OpenGL to do a lot of the screen rendering which opens up a lot of possibilities for cool-new-shiney-bits later on.

Matthew: What’s the competition for Xibo?

Alex and Dan: There’s huge competition in the digital signage market place, however we’re frequently told that our offering is better than a lot of the commercial signage applications people have used before. In terms of Open Source competition, there is the Concerto Signage Project who are based at the Rensselaer Polytehnic Institute and released under the GPL v2.

What we’re hoping to bring to the DS market place is a Free solution translated in to many languages with a thriving community of contributors working together to make Xibo the best it can be. An example is the new Xibo Layout Exchange. Here you can download contributed artwork for use in the Xibo system but eventually we plan to allow people to share whole bundles of content.

Matthew: Are you building a business around Xibo?

Alex and Dan: The Xibo application is Free and we have no plans to change that. There are many options available to monetize Xibo (content creation, hosting, support, custom development etc) but there’s nothing in the pipeline at the moment.

Matthew: Why did you select the AGPL 3 as Xibo’s licence?

Alex and Dan: Xibo is written with SaaS (Software as a Service) in mind. We’re very happy for businesses to take Xibo, rebrand it and sell it as a service to their customers, but the freedom needs to be two-way. We wanted a license that ensured that modifications these companies made would be accessible to the end user, for the greater good of the project. AGPLv3 offers us those things, while being compatible with a lot of existing library code.

Matthew: Is there a community of people interested in developing an open source signage system?

Alex and Dan: Xibo currently has two main developers. We’ve had code contributions from a few other people to date, but there’s a fair learning curve which presents a hurdle for prospective developers. We’re working towards a more modular architecture which will allow people to develop plugins to the server and client to extend it’s functionallity, which should lower the barrier to entry significantly.

There’s an active support community already in the Forum and in Launchpad Answers. We’re also taking art submissions for the Layout Exchange, and there is already a huge list of blueprints in Launchpad for future development, contributed by many people.

At least one of the developers is planning to be at LugRadio Live 2009 and Oggcamp in Wolverhampton later this year — just as a visitor, we aren’t exhibiting, but they’ll be suitably dressed so come over and say hello!

Matthew: Heh, see you there. So, why did you choose Launchpad?

Alex and Dan: We have used Sourceforge and Subversion before, but we wanted to use Bazaar for the RCS and Launchpad gave us Bazaar hosting and a good bug tracker. The translations support has been a huge bonus as has Blueprints and Answers. It rolls almost everything we needed in to one convenient package. It also acts as an OpenID provider so we can give the community secure access to edit the wiki and authenticate with the forum.

Matthew: What have you found particularly useful in Launchpad?

Alex and Dan: I think possibly the single most useful feature is merge request tracking – allowing us to fix bugs in individual branches and then queue the fixes up for merging later on. Answers has been great for doing user support – especially where there’s four or five of us helping someone it gives a good overview of which issues are outstanding, and a clear progression to a bug if needed. The bugtracker integration with bzr is great too for keeping track of where fixes went.

Matthew: What would you like to see added to Launchpad?

Alex and Dan: In the early days it would have helped us a lot if Launchpad had provided a wiki too. Now we’ve got our own system in place migrating over makes less sense though. I’d love to see a “Convert to Blueprint” button in Answers, as we get a lot of people making feature requests there.

Matthew: Thanks both!