November 9, 2015

Launchpad blog
Launchpad blog
Launchpad news, October 2015

Here’s what the Launchpad team did in October.


  • Ensure that enabled and restricted processors are left untouched when submitting Archive:+edit (#1501519)
  • Describe merge proposal listings in the second person (#991945)
  • Increase Git backend communication timeout to 30 seconds (from 5), as detecting merges can be quite slow on large repositories (#1502748)
  • Link directly to the recipe/snap in the one-recipe/snap case, rather than redirecting from +recipes/+snaps views (#891918)
  • Add GitRepository:+subscription traversal (#1503749)
  • Send notification mail when a merge proposal is superseded (#716169)
  • Add Branch:+dependent-merges and GitRef:+dependent-merges views (#496056)
  • Parse extended header lines in git diffs correctly (#1510337)

Additionally, we’ve added webhook support for merge proposals, made a few more UI improvements, and enabled webhooks for beta testers.  Feedback welcome!

Snap packages

  • Allow snap package owners to enable/disable unrestricted processors (matches the change made for PPAs last month)

Soyuz (package building)

  • Launchpad now accepts uploads of binary packages from builders with file modification times up to 24 hours in the future (rather than 8) or as early as 1975 (rather than 1984), to match Debian
  • Notifications are no longer generated for package copy jobs that in fact end up being no-ops

Also, as previously announced, we and our sysadmins have been putting a lot of work into extending ScalingStack to other architectures and making it more reliable, and as a result we can now offer general access to PPAs for the ppc64el architecture.

Beta test: webhooks

If you are a member of Launchpad’s beta testers team, you can now try out webhooks for Bazaar branches and Git repositories. These can be used to set up integration with external sites for various purposes, such as running CI jobs or publishing documentation. We expect to open this up to all Launchpad users soon, but in the meantime please do file a bug against Launchpad itself if you encounter any problems.

See our webhooks documentation for more details.

Update: as of 2015-11-20, this feature is enabled for all Launchpad users.

October 27, 2015
PPAs for ppc64el

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs.  Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time.  Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder.  Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine.  Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build.  (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system.  Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build.  Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will.  To do this, visit the main web page for your PPA (which will look something like “<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out.  This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment.  arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available.  We’ll post further announcements when further architectures are unrestricted.

October 4, 2015
Launchpad news, September 2015

October already! As the leaves start to turn red here in the northern hemisphere, here’s a brief summary of what we did in September.


  • BugTask:+addcomment’s title doesn’t duplicate the bug number (#1323808)
  • The duplicates portlet of bug tasks no longer links to invisible private bugs and now uses the correct sprite for each bug (#1443418, #1465880)
  • Show hidden bug comments to their owners (#1391394)


  • Line numbers in merge proposal preview diffs are now unselectable (#1483925)
  • Branch revision listings link more sensibly to merged branches and merge proposals (#711647)
  • Git repositories and refs have more sensible breadcrumb links (#1466271)
  • Source package recipe builds now include the distribution series in their titles (#1491336)
  • Allow archive owners to cancel their recipe builds, and make recipes use the normal cancellation infrastructure (#624630)
  • Make it possible to retry superseded builds, since there are situations where they can become unsuperseded (#444030)
  • The index page for Git refs is more useful
  • Fix crashes on various +activereviews pages when there are no active reviews (#1499744, #1501134)
  • Precache branch permissions in Branch.landing_candidates, fixing API timeouts (#1500576)
  • Add webhook support for Bazaar branches, currently enabled on qastaging though not yet on production


  • Team membership notifications now honour the “Include filtering information in email footers” setting, and have rationale information in the headers and footer (#296889, #508897)
  • There is a new X-Launchpad-Message-For mail header (or Launchpad-Message-For in expanded footers) giving just the name of the person or team directly subscribed to the notification (#1493844)
  • Treat “me” in person-or-team contexts in mail handlers as the current user (#340397)

We now consider the “Gmail filtering improvements” work complete.  Let us know if there are further categories of mail sent by Launchpad that you’re finding difficult to filter using Gmail.


  • Distribution index pages hide links to disabled features (#80315, #257627)
  • Set consistent colour for the “Opinion” bug task status (#648645)
  • Blueprints on milestone pages have icons again and are sortable (#1354387)
  • Old user-to-user email database references no longer prevent changing team privacy (#1498497)

Snap packages

  • Snap packages now have edit, administer, and delete views
  • Branches, Git repositories and refs, people, teams, and products now have snap package listing views
  • Members of ~launchpad-snap-builders can create snap packages in the web UI based on Bazaar or Git branches
  • Snap package owners can request builds in the web UI

Soyuz (package building)

  • Stop showing the confusing .dsc component on source package index pages (#521722)
  • Export archive deletion on the API (#814633)
  • Allow overriding the build version for live filesystem builds (#1496074)
  • Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin
  • The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)
  • Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it
  • Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available
  • Enabling a test rebuild archive no longer times out (#1500973)

September 2, 2015
Launchpad news, August 2015

Here’s a summary of what the Launchpad team got up to in August.


  • Webhook support for Git repositories is almost finished, and only needs a bit more web UI work (#1474071)
  • The summary of merge proposal pages now includes a link to the merged revision, if any (#892259)
  • Viewing individual comments on Git-based merge proposals no longer OOPSes (#1485907)

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail.  The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do.  Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

  • Bug notifications and package upload notifications now honour the “Include filtering information in email footers” setting (#1474071)
  • Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)
  • Recipe build notifications now include an X-Launchpad-Archive header (#776160)
  • Question notification rationales are now more consistent, including team annotations for subscribers (#968578)
  • Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

  • Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated
  • Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested
  • Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting
  • All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity
  • We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs
  • We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)


  • The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)
  • We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically
  • Your user index page now has a “Change password” link (#1471961)
  • Bug attachments are no longer incorrectly hidden when displaying only some bug comments (#1105543)

August 2, 2015
Launchpad news, July 2015

Here’s a summary of what the Launchpad team got up to in July.


  • We fixed a regression in the wrapping layout of side-by-side diffs on (#1436483)
  • Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August


  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

July 29, 2015
Improved filtering options for Gmail users

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively.  We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders.  Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject.  Figuring out what to do about this has been tricky.  Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:

Screenshot of email configuration options

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer.  Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable.  You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

Screenshot of Gmail filter dialog with

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures.  We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications.  If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

July 9, 2015
Launchpad news, April-June 2015

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.


Of course, you don’t get to release a major new feature and then have everything be perfect.  In fact, we’d released it pretty much as soon as we had a minimum viable product, so we knew there was plenty more to do.

Now that the basics work reasonably well, we’ve been focusing on tying off loose ends, and on getting to the point where Launchpad itself can be self-hosted in Git: this is partly because most of the things we need to do for that are relevant to many other projects considering a migration, partly because that ensures that any problems with it will affect Launchpad developers directly, and partly because this will allow us to trim a very large amount of database cruft associated with the representation of Launchpad’s own branches.

Here’s a brief changelog since our initial public announcement:

  • Leaving the “Target reference path” field empty when proposing a Git-based merge proposal no longer crashes (#1451068)
  • Git-based merge proposals can be deleted
  • Pushing to private repositories no longer hangs (#1451107)
  • The backend now stores reflogs so that we can help users recover from mistakes
  • Pushing a new default repository for a project now sets the repository owner to the project owner rather than to the person performing the push
  • The project +sharing view now handles Git repositories
  • Branches of a project default repository are now shown on the project’s code page, although we expect to be making significant further improvements in this area soon
  • Individual Git commits under “Recent Commits” now link directly to a full view of the corresponding commit
  • Merge proposal listings now cover Git (#1453020)
  • Branches in private repositories now show a privacy banner (#1457553)
  • Repositories can now be deleted (#1456583)
  • Private repositories are now browseable and show a privacy banner in the browsing interface
  • The URLs for Git-based merge proposals have moved to be directly under the source repository rather than under the source branch, so that historical MPs can still be accessed after deleting the branch (#1456589)
  • Launchpad now sends mail for Git-based merge proposals
  • There are various new Git repository listing views, chiefly for projects, packages, and people
  • Repositories now have a complete edit form allowing you to rename them, move them between targets, and set their default branch (“HEAD”) (#1456625)
  • We’ve exported a few more bits and pieces on the webservice API to support Git merge robots
  • We’ve brought various bits of automation for Git-based merge proposals up to par with their Bazaar equivalents: when a Git branch is updated, Launchpad now automatically updates the preview diffs for any MPs where that is the source branch, and detects whether any MPs where that is the target branch can now be marked as Merged
  • The Code settings view for projects now allows you to set the project’s default Git repository and to choose whether the default revision system for the project is Bazaar or Git

We’ve started on support for webhooks, and hope to be able to tell you more about this soon. Git-to-Git repository mirroring and native Git recipes are also frequent requests that we know are important to people, and we hope to get to those soon.

Other code review changes

Many of the changes needed for Git touched on our code review infrastructure, but we’ve been making some other improvements there too.  Launchpad has supported “inline comments” on merge proposals for a year or so now, allowing you to comment on parts of a diff without having to manually quote it; but the feature hadn’t been quite finished and lots of people had nits to pick with it.  We’ve made a couple of improvements here lately:

  • Preview diffs in merge proposals now have keyboard navigation to move between files, diff hunks, and inline comments
  • Links to diffs now include a per-file diffstat under an expander triangle
  • The mails sent when a reviewer adds inline comments now only include relevant parts of the diff context rather than the entire diff, making them much easier to read and making it harder to miss comments (#1334577)

We threw in a couple of bonus features here as well.  You can now see side-by-side diffs for merge proposals as well as unified diffs (#917502): there’s a link at the top of each preview diff allowing you to switch it to side-by-side mode (we may add a user preference for this later).  And resubmitting a merge proposal now automatically preserves the commit message (#676769).

Package build infrastructure

When we first added support for building debug symbol packages (ddebs) in Ubuntu, we handled the build and publication side of things with a temporary hack involving a custom hack to our sbuild fork that stashed the ddebs on the builder for a while, and a job that periodically fetched them to As is the way of temporary hacks, it proceeded to stay that way for eight and a half years. It mostly worked, but was rather fragile, contributed to tying us to an old sbuild, wasn’t very extensible to PPAs, and couldn’t work with virtualised builders where all the builder’s state is reset before every build.

A couple of years ago we put together the Launchpad changes required to store ddebs straightforwardly in Launchpad instead, but weren’t able to deploy this at the time due to needing better librarian infrastructure. We finally got all this cleared out in April and deployed the new ddeb publishing mechanism. The main user-visible effect of this is that isn’t liable to lose packages any more, and ddebs are also available from Launchpad build pages.

Having done this, we were able to upgrade from our 2004-era fork of sbuild to a modern version, which fixed a number of long-standing bugs.

All architectures are now optional for PPAs (#1244868), so, for example, a PPA that only needs to support armhf can save builder cycles by disabling amd64 and i386. At the moment this can only be done by Launchpad administrators, so ask a question on Launchpad if you think this would be worth doing for any of your PPAs.

We rewrote the way that we install build-dependencies for source package recipes. This allows recipes to handle the “:native” qualifier in build-dependencies which is used in some places as part of supporting cross-building.

The next step in this modernisation programme will be to move all builds into ScalingStack and decommission the corresponding bare-metal builders, which we’ll be starting soon and working on an architecture-by-architecture basis as that cloud gains support for them. In the short term, this will give us better build capacity on the migrated architectures; in the longer term, it will let us support PPA builds on more architectures, and erase the cumbersome distinction between “non-virtualised” and “virtualised” builders. More news on this as it happens.


We upgraded to significantly more modern database servers earlier this year, which meant that a lot of hitherto difficult timeout problems suddenly either disappeared or became tractable. There are still some bad queries and a few mysterious problems, but as a general trend our timeout rates are very significantly down from where they were six months ago. We have some more general plans in this area, and will continue to spot-fix bad pages as they show up and as time permits.


We have lots more to do across the whole application, but still have a rather limited number of developers. If any of this kind of thing sounds interesting and you’d like to help, you can!

May 1, 2015
Git code hosting beta

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

  • push Git repositories to Launchpad over SSH
  • clone repositories over git://, SSH, or HTTPS
  • see summary information on repositories and the branches they contain in the Launchpad web UI
  • follow links from the Launchpad web UI to a full-featured code browser (cgit)
  • push and clone private repositories, if you have a commercial subscription to Launchpad
  • propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

  • useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
  • RSS feeds
  • mirroring
  • webhooks
  • an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Helping out

This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

October 19, 2014

Curtis Hovey
Sinzui » Sinzui
Reverting Core Storage Physical Volumes

I after upgrading OS X to Yosemite, rEFInd did not start. Using the Option key while rebooting, I selected my Ubuntu Trusty partition, which booted with a warning that the MacintoshHD couldn’t be mounted.

During the upgrade, the HFS+ partition where OS X resides was converted to a logical volume of Apple Core Storage. I believe this is to support encryption, but I declined the option during the upgrade. The Disk Util app in OS X will not show this change, but the diskutil command will. The fix is trivial from the OS X terminal.

The diskutil list command will reveal which partition was convert to a logical volume on the physical disk.

diskutil list
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         175.6 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
   4:       Microsoft Basic Data                         70.3 GB    disk0s4
   5:                 Linux Swap                         4.2 GB     disk0s5
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *175.3 GB   disk1
                                 Logical Volume on disk0s2

Your disk partitions for dual booting Ubuntu and OS X may be different. I know there was no logical volume before I started the upgrade. I restored the APPLE_HFS partition to the real disk0s2 with the revert option

diskutil corestorage revert /dev/disk1

When I rebooted, rEFInd presented me with choice of OS X and Ubuntu. Ubuntu mounted with MacintoshHD when I signed in. All fixed.

June 12, 2014
Migrating juju to HP Cloud’s Horizon

HP Cloud retired its old regions this month. If you use juju on HP Cloud and your account was created before this year, you may need to update your juju config and possibly complete your project setup in Horizon.

There are two changes you must make to each HP Cloud environment listed in environments.yaml. Set the region option to “region-a.geo-1″ or “region-b.geo-1″, which are US West and US East respectively. The new regions do not automatically assign public IPs. The use-floating-ip option must be set to “true” to tell juju to request a public address.

use-floating-ip: true
region: region-a.geo-1

Note that juju does not yet release the public address when you destroy the machine, You need to use the nova command line tools or the Horizon console > Project > Compute > Access & Security > Floating IPs to release them. We will release a fix for “Floating IPs are not recycled in OpenStack Havana” soon.

If you haven’t used these regions before, you needs to register your public SSH keys and create a default network for each one you use. Visit Horizon console > Project > Compute > Access & Security > Key Pairs. Most people migrating just copied their public keys to the new region. Visit Horizon console > Project > Networks > Network and verify or create a default network, something like

June 10, 2014
Building trans-cloud environments with juju

The Juju QA team uses juju to build the CI services that test juju. We have built 3 CIs in fact, because juju makes it easy to move services to the cloud of our choice. Our choice is driven by which cloud provides the best combination of resources, but CI has grown beyond what one cloud can provide. We want CI in all clouds to test with the resources they provide. While juju doesn’t support cross environment relations, that hasn’t stopped the QA team from building an hybrid environment that straddles two clouds and a private network.

The steps to provision a machine on another network are codified in our add-remote-machine.bash script. The Juju QA team uses this script to add physical machines, machines in kvm, and machines launched in other clouds into our Juju environment. Once a machine is registered with a juju environment, any charmed service can be deployed to it. Adding a machine to a open network is painless.

add-remote-machine.bash juju-ci3 my-keys/juju_ci_rsa
    |                          |          |              |
  script                     env-name   ssh-key   private-ip

Adding machines on restricted networks may require firewall egress changes.

Provisioning the best resources for services

Juju provides two essential devops features:

  • A cloud-agnostic way to provision machines. Juju supports Azure, EC2, Joyent, MAAS, and OpenStack-based clouds like HP Cloud.
  • A trivial means to deploy configured services to machines. I can put Jenkins slaves and web applications into production in minutes.

The mechanism to deploy services is so valuable, that juju provides a command to add existing machines to the environment. Existing machines often have special resources not provided by a simple cloud image. Using placement, I can deploy one or more services to each machine. For example, to add a machine that I want Jenkins to use, I would issue these commands to register machine number 2 in my environment, then deploy a jenkins slave to it, and finally configure the existing jenkins master to send work to it:

juju add-machine ssh:ubuntu@
juju deploy --to 2 jenkins-slave ppc64-slave
juju add-relation jenkins-master ppc64-slave

Provisioning remote machines

The problem is that juju wants a private address when provisioning a machine. The solution to building hybrid environments composed of machines from other networks was understanding what network resources juju requires during and after provisioning. First, there are three parts to juju:

  • The juju client (juju command line or juju-gui) that issues commands
  • The juju state-server that manages the machines and services in the environment
  • The many juju agents (1 for each machine and service) that asks the state-server for tasks

During the act of provisioning (bootstrap or add-machine) the client acts as a bridge between the state-server and the remote machine. The private address is thus private to the host that the client is running on! Once the agent is running on the remote machine, it will talk to the state-server using the state-server’s private DNS name. The state-server doesn’t really know where the remote machine is, and it cannot access it.

Since I have ssh access to all the machines via their private address, I can add them to the juju environment. Even when I only know a machine’s public address, I can add an ssh rule that maps a random private address to the public host. As for helping the agent talk to the state-server, a single addition to the agent’s /etc/hosts is required to map the state-server’s public IP address to the private DNS name.

The script checks the required connectivity. It tries to verify common addresses that juju will use, such as the location of the environment’s private container, the server images, and juju tools. It is not authoritative. It will ask me to verify some connections that are not explicitly defined in the environment. It doesn’t know the network requirements of the services that will be deployed to the machine. This isn’t an issue for machines on open networks. Several of the Juju CI machines are on restricted networks, and I asked the IS department to allow egress to the required addresses and ports. We may improve the script’s verification support as we extend the environment to HP Cloud and Joyent.

As for how Juju makes CI easier, that is the subject or a future post.

April 21, 2014
Working with Juju, Ubuntu 12.04 Precise, and 14.04 Trusty

Juju 1.18.x is the first Juju that is aware of two Ubuntu LTS releases.
In the past, when bootstrapping the state-server or deploying a charm,
Juju selected Ubuntu 12.04 Precise when the serie was not specified. If
you care about the version of Ubuntu used, then you want to specify the

You can specify the series of the state-server and set the default charm
series by adding “default-series” to environments.yaml. Setting
“default-series” is the only way to specify the series of the
state-server (bootstrap node). For example, if your local machine is
Ubuntu 14.04 trusty, but your organisation only supports Ubuntu 12.04
Precise in the cloud, you can add this to your environment in

default-series: precise

There are many ways to specify the series to deploy with a charm. In
most cases you don’t need to. When you don’t specify a series, Juju
checks the environment’s “default-series”, and if that isn’t set, Juju
asks the charm store to select the best series to deploy with the charm.
The series of the state-server does not restrict the series of the
charm. You can use the best series for a charm when deploying a service.

When working with local charms, Juju cannot fall back to the charm
store, it falls back to the environment’s “default-series”. You must
specify the series in the environment or when deploying the charm. If
your environment is running, you can add “default-series” like so:

juju set-environment default-series=precise

These commands choose Ubuntu 12.04 Precise when “default-series” is set to “precise” in the environment:

 juju deploy cs:precise/mysql
 juju deploy precise/mysql
 juju deploy mysql
 juju deploy local:precise/mysql
 juju deploy loca:mysql

November 16, 2013
Restoring network to lxc and juju local-provider

I have experienced two cases where lxc containers stop working or new containers never work because they cannot join the network. My LTS container started, but without network, there were a lot of things I couldn’t do in it. In the case of Juju, I couldn’t deploy new local services. The unit agent status was stuck in PENDING.

You can verify the network is broken by fancy listing the lxc containers you started either by lxc-start or juju deploy:

sudo lxc-ls --fancy

<container> RUNNING -    -    NO

This shows that the running container doesn’t have IPV4 or IPV6 networks.

For each existing container that needs fixing, you need to install the new dhcp packages, but without a network, you cannot do it from a running container. Instead chroot can be used to update the container’s root file system.

cd /var/lib/lxc/<container>/rootfs/
sudo chroot ./
sudo apt-get update
apt-get install isc-dhcp-common isc-dhcp-client

To ensure that new containers work, you need to clear the lxc cache. The lxc image cache was built when you created or deployed your first container. The images there are probably more than a few months old.

sudo ls -lh /var/cache/lxc/precise/
sudo ls -lh /var/cache/lxc/cloud-precise/
sudo rm -r /var/cache/lxc/*

Lxc will get the new images the next time it needs to create a container. The first call will be slower to complete since it will build a fresh cache.

October 28, 2013
Restoring grub after OS X Mavericks

I had a scary moment after I updated my dual boot MacBookAir to Mavericks. Refit started Ubuntu, but grub was showing me the rescue prompt. This was a different misadventure from my upgrade to Mountain Lion; the partitions were there, but grub was lost.

I use refit to manage dual boot. I remembered from past experience that I needed to use the partition tool from the refit boot screen to sync the GPT and EFI tables. The task was done in seconds.

I knew Ubuntu was on the 4th partition (suggested by the GPT output from the sync), but to be sure I listed the partitions, then listed the 4th one.

ls (hd0,gpt4)/boot

and I saw the boot images and a grub dir. I then assumed, all my data was in place, but the MBR was lost during the Mavericks install. To start Ubuntu, I typed:

set root=(hd0,gpt4)
set prefix=(hd0,gpt4)/boot/grub
insmod normal

Ubuntu started up as normal. The grub screen was identical to my last boot. I was confident grub’s configuration was fine, I just needed to restore grub to the MBR. I opened a terminal after logging in and typed:

sudo grub-install --force /dev/sda

With hesitation, I rebooted. All was back to normal.

September 2, 2013

Launchpad blog
Launchpad blog
Launchpad build farm improvements

We’ve made a number of improvements to the Launchpad build farm in the last month, with the aim of improving its performance and robustness.  This sort of work is usually invisible to users except when something goes wrong, so we thought it would be worth taking some time to give you a summary.  Some of this work was on the Launchpad software itself, while some was on the hardware.

(To understand some of the rest of this post, it’s useful to be aware of the distinction between virtualised and devirtualised builders in Launchpad.  Virtualised builders are used for most PPAs: they build untrusted code in a Xen guest which is initialised from scratch at the start of each build, and are only available for i386, amd64, and a small number of ARM builds by way of user-mode QEMU.  Devirtualised builders run on ordinary hardware with less strict containment, and are used for Ubuntu distribution builds and a few specialised PPAs.)

ARM builders have been a headache for some time.  For our devirtualised builders, we were using a farm of PandaBoards, having previously used BeagleBoards and Babbage boards.  These largely did the job, but they’re really a development board rather than server-class hardware, and it showed in places: disk performance wasn’t up to our needs and we saw build failures due to data corruption much more frequently than we were comfortable with.  We recently installed a cluster of Calxeda Highbank nodes, which have been performing much more reliably.

It has long been possible to cancel builds on virtualised builders: this is easy because we can just reset the guest.  However, it was never possible to cancel builds on devirtualised builders: killing the top-level build process isn’t sufficient for builds that are stuck in various creative ways, and you need to make sure to go round and repeatedly kill all processes in the build chroot until they’ve all gone away.  We’ve now hooked this up properly, and it is possible for build daemon maintainers to cancel builds on devirtualised builders without operator assistance, which should eliminate situations where we need urgent builds to jump the queue but can’t because all builders are occupied by long-running builds.  (People with upload privileges can currently cancel builds too, which is intended mainly to allow cancelling your own builds; please don’t abuse this or we may need to tighten up the permissions.)  As a bonus, cancelling a build no longer loses the build log.

Finally, we have been putting quite a bit of work into build farm reliability.  A few problems have led to excessively long queues on virtual builders:

  • Builders hung for some time when they should have timed out, due to a recent change in su; this is now fixed in the affected Ubuntu series.
  • Xen guests often fail to restore for one reason or another, and when this happened builders would fail in ways that required an operator to fix.  We had been dealing with this by having our operators do semi-automatic builder fixing runs a few times a day, but in recent months the frequency of failures has been difficult to keep up with in this way, especially at the weekend.  Some of this is probably related to our current use of a rather old version of Xen, but the builder management code in Launchpad could also handle this much better by trying to reset the guest again in the same way that we do at the start of each build.  As of this morning’s code deployment, we now do this, and the build farm seems to be holding up much more robustly.

This should make things better for everyone, but we aren’t planning to stop here.  We’re intending to convert the virtual builders to an OpenStack deployment, which should allow us to scale them much more flexibly.  We plan to take advantage of more reliable build cancellation to automatically cancel in-progress builds that have been superseded by new source uploads, so that we don’t spend resources on builds that will be rejected on upload.  And we plan to move Ubuntu live file system building into Launchpad so that we can consolidate those two build farms and make better use of our available hardware.

June 25, 2013

Curtis Hovey
Sinzui » Sinzui
Managing Juju Charm Versions

It is difficult for Juju charm authors to support forked charms. dev-ops often don’t realise they are forking a charm when they add files that they want the charm to deploy. We cannot assume that the deployed charm’s revision (or the version control revision number) will ever match what the author released. I use Bazaar tags to mark the versions I support and help identify the true version of a deployed charm.

Juju uses the charm’s “revision” file to store the version. The number in this file must be incremented to deploy additions or changes to the charm. Upgrading a deployed charm requires a higher revision number. Organisations commonly create a local charm repo (managed by a version control system like Bazaar) to freeze the versions they deploy. These known version are stable; this is a best practice when working with charms that change often. The local branch will get commits that are not in the branch published at Launchpad. The revision numbers are meaningless between the two branches.

I tag my charm branches with the charm revision number. When a dev-op forks my charm to add it to the local charm repo, we can compare branch tags. For example, I tagged lp:~charming-devs/charms/precise/elasticsearch/trunk r39 as elasticsearch-29 to match the number in the revision file. I can ask the dev-op which tags the local branch has. I can quickly identify missing releases. I can ask for a diff from the tag I last released to help diagnose problems with the changes.

For example, several issues were reported about my elasticsearch charm. I had addressed the problems months before. I suggested the dev-ops upgrade to the latest version. Since their version numbers were higher than my versions, they did not upgrade. Tags are more precise. I can suggest the dev-ops merge “-r tag:elasticsearch-29″.

Another case, charms may need to change often if the application is deploys is under active development. A recent deploy of charmworld failed; the charm upgrade errored. The dev-op and I couldn’t reconcile the version numbers between what was deployed and what I released. We had to read a diff of the charm’s directories and files to discover the deployed charm was missing several releases–the charm was incompatible with the deployed code. If I had tagged my release, we could have identified the issue in minutes.

June 6, 2013
Closing milestone bugs using Launchpad’s API

A few years ago I wrote a contrib script for Launchpad’s launchpadlib called ‘’ which attempted to close (aka mark them ‘fix released’) all of your bugs in a project that were targeted to a particular milestone.

For various reasons it grew out of date and when I needed to use it recently, it didn’t work!  Long story short, I just fixed it up and added a couple of new features:

  • You can optionally close just your own bugs, or all the bugs in the milestone
  • You can search for bugtasks targeted against a series in your project (these are not normally picked up when searching in a project’s milestone)

You can grab the code here:

bzr branch lp:launchpadlib


April 24, 2013
Error handling in Go

There’s been a debate raging in some corners of the internet lately about how superior Go‘s error handling is to other languages.  I am going to address some of the points made, here:

Claim 1: It’s impossible to ignore errors in Go, they are “in your face”

This is patently false.  Take this example:

fmt.Println("Hello world")

Pretty innocuous wouldn’t you say?  Well let’s take a look at the language documentation for fmt.Println:

// Println formats using the default formats for its operands and writes to standard output.
// Spaces are always added between operands and a newline is appended.
// It returns the number of bytes written and any write error encountered.
func Println(a ...interface{}) (n int, err error)

So Println can return an error!  Where did we check it?  Well, we didn’t.  Any other claims that it’s OK to ignore it in this case further strengthen my argument.

Some will say that it’s a deliberate choice to ignore the error and I deserve all I get. Well, was it? I didn’t even know that Println returned an error until I looked at the documentation (and who is going to do that for Println?). And that’s the point, if I need to look at the documentation to see that it can return an error, then if I am using a language that raises exceptions I will have also seen its documentation about how it deals with errors.

You could even argue that an exception is superior in this case.  With Go, the code will march on regardless, oblivious to the fact that Println failed.  With exceptions, it’ll fail and show you exactly where it failed.

The language will error at compile time if you try to ignore an error returned as a second value and you only take the first.  But this is trivially bypassed by assigning it to _, which when reading code is easily missed compared to the exception style of “catching then dropping”, because Go itself encourages this style of assigning to _ with its own range statement as a deliberate way of ignoring things that the language is trying to force you to see.

So really in both cases, ignoring the error doesn’t really stand out as wrong.

Here’s a concrete example in Go I was recently shown:

w := bufio.NewWriter(os.Stdout)
for _, name := range ListAll(conf) {
    fmt.Fprintln(w, name)

As you can see, the caller completely forgot to check the error returned from Fprintln and Flush and there would be no compiler warning about it.

Claim 2: Exceptions teach developers to not care about errors

Citing an example where someone didn’t catch an exception and the code consequently blew up is really not a good example of this claim.  It’s a bug, for sure and you get a full traceback of your error in the resulting exception, which is handy.  You go away and fix it quickly based on that info.

If I am in the same situation with Go and I ignore a returned error from a function, at some point (which is likely to be nowhere near the place where the error occurred) my code will blow up.  I’ll have to run up the debugger to try and find out where it really occurred though.

Because unused variables in Go are a compile-time error, it’s actively discouraging you from assigning the result of the function to a variable (or you can deal with it, of course).  For anyone who’s not read the full documentation for a function call or missed its return value (we’re all human) as I said above – you’re not even going to notice that you missed it.

Based on this, I can see no difference at all that suggests one way or the other teaches developers to not care about errors.  Developers do care about errors, really, but bugs creep in however careful you are.  And when they do, I’d rather have a decent indication of where the bug is.

Other parts of error handling that I dislike

When you look at the average Go program, you will see a lot of this:

if err != nil {
  return nil, err

This is the recommended way of error handling in in Go.  But this is not error handling, it’s error propagation.  In nearly all languages there will arise situations where in well-factored code you have a low-level error that you need to pass right back up to the entry point for the caller. That means you need this error propagation code in every single place where you check for errors.  There’s no syntactic sugar, just the same three lines everywhere.

For me, this vastly decreases the readability of the code. This is where exceptions excel because inside my own library I can factor the bejeesus out of it into many small functions and if I need to return an error, I just catch a lower-level exception in the top-level function and return something else.  You can do this in Go with a panic(), but it seems to be discouraged.  Panic() feels almost exactly like using exceptions, only the syntax is worse. If Go’s style is to encourage people to handle errors like this, it needs the sugar.


Many people might think that I completely hate Go’s error handling from this post.  That’s not strictly true – I don’t hate it, I just think it can be improved.  I challenge assumptions that I see which state that Go’s error handling is superior in some way, when as far as I can see it’s not that different from other languages in terms of usefulness.

Go is clearly in its infancy.  Most languages will have started out with youthful enthusiasm and realised that some change was needed.  These languages are the successful ones where developers enjoy coding in it and feel productive.  I hope that Go embraces change as it matures and attracts more developers.

I welcome comments on this post – unlike some people I won’t censor them or delete ones I can’t argue with (unless they are outright abusive and use foul language, this is a family blog!).

December 4, 2012

Launchpad blog
Launchpad blog
Private Projects and Private Blueprints leave beta

Today, the Private Projects and Private Blueprints features on Launchpad are leaving beta. These features are available now for use by all Launchpad users. Private Blueprints was started as part of the Private Projects work, with the end goal in mind of truly private projects on Launchpad. Private Projects was described in its beta announcement like this:

When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.

When you create a proprietary or embargoed project on Launchpad, all of the sharing policies for your project will be set correctly for you. This means that if you start your project as a proprietary project, your branches, bugs, and blueprints will be created proprietary by default. Answers and translations are not available for private projects.

A commercial subscription is required to use private projects, but any user who creates a proprietary or embargoed project on Launchpad will receive a 30 day trial commercial subscription. Launchpad users with existing commercial subscriptions can convert a public project to proprietary or embargoed by changing the information type in the project’s settings. You may have some work to do on your project before you can transition to a private information type — for example, disable answers if you have that app enabled for your project — but Launchpad will block the change and tell you what needs to happen before you can switch to a private information type.

Users should be aware, though, that if your project has been listed on Launchpad publicly until now, then search engines know of its existence already. If you want a proprietary project that no one can learn of its name, you should create a new project on Launchpad. Transitioning a public project to private allows you to keep your series and milestones private going forward, but users may have already been able to discover the existence of the project since it was public already.

We are happy to make truly private projects available for all users on Launchpad. If you run into any issues, please file a bug against Launchpad or ask for help in #launchpad on Freenode.