August 2, 2015

Launchpad blog
lp-blog
Launchpad blog
Launchpad news, July 2015

Here’s a summary of what the Launchpad team got up to in July.

Code

  • We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)
  • Various code pages now have meta tags to redirect go get to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the go tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

July 29, 2015
Improved filtering options for Gmail users

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively.  We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders.  Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject.  Figuring out what to do about this has been tricky.  Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:

Screenshot of email configuration options

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer.  Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable.  You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

Screenshot of Gmail filter dialog with

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures.  We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications.  If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

July 9, 2015
Launchpad news, April-June 2015

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.

Git

Of course, you don’t get to release a major new feature and then have everything be perfect.  In fact, we’d released it pretty much as soon as we had a minimum viable product, so we knew there was plenty more to do.

Now that the basics work reasonably well, we’ve been focusing on tying off loose ends, and on getting to the point where Launchpad itself can be self-hosted in Git: this is partly because most of the things we need to do for that are relevant to many other projects considering a migration, partly because that ensures that any problems with it will affect Launchpad developers directly, and partly because this will allow us to trim a very large amount of database cruft associated with the representation of Launchpad’s own branches.

Here’s a brief changelog since our initial public announcement:

  • Leaving the “Target reference path” field empty when proposing a Git-based merge proposal no longer crashes (#1451068)
  • Git-based merge proposals can be deleted
  • Pushing to private repositories no longer hangs (#1451107)
  • The backend now stores reflogs so that we can help users recover from mistakes
  • Pushing a new default repository for a project now sets the repository owner to the project owner rather than to the person performing the push
  • The project +sharing view now handles Git repositories
  • Branches of a project default repository are now shown on the project’s code page, although we expect to be making significant further improvements in this area soon
  • Individual Git commits under “Recent Commits” now link directly to a full view of the corresponding commit
  • Merge proposal listings now cover Git (#1453020)
  • Branches in private repositories now show a privacy banner (#1457553)
  • Repositories can now be deleted (#1456583)
  • Private repositories are now browseable and show a privacy banner in the browsing interface
  • The URLs for Git-based merge proposals have moved to be directly under the source repository rather than under the source branch, so that historical MPs can still be accessed after deleting the branch (#1456589)
  • Launchpad now sends mail for Git-based merge proposals
  • There are various new Git repository listing views, chiefly for projects, packages, and people
  • Repositories now have a complete edit form allowing you to rename them, move them between targets, and set their default branch (“HEAD”) (#1456625)
  • We’ve exported a few more bits and pieces on the webservice API to support Git merge robots
  • We’ve brought various bits of automation for Git-based merge proposals up to par with their Bazaar equivalents: when a Git branch is updated, Launchpad now automatically updates the preview diffs for any MPs where that is the source branch, and detects whether any MPs where that is the target branch can now be marked as Merged
  • The Code settings view for projects now allows you to set the project’s default Git repository and to choose whether the default revision system for the project is Bazaar or Git

We’ve started on support for webhooks, and hope to be able to tell you more about this soon. Git-to-Git repository mirroring and native Git recipes are also frequent requests that we know are important to people, and we hope to get to those soon.

Other code review changes

Many of the changes needed for Git touched on our code review infrastructure, but we’ve been making some other improvements there too.  Launchpad has supported “inline comments” on merge proposals for a year or so now, allowing you to comment on parts of a diff without having to manually quote it; but the feature hadn’t been quite finished and lots of people had nits to pick with it.  We’ve made a couple of improvements here lately:

  • Preview diffs in merge proposals now have keyboard navigation to move between files, diff hunks, and inline comments
  • Links to diffs now include a per-file diffstat under an expander triangle
  • The mails sent when a reviewer adds inline comments now only include relevant parts of the diff context rather than the entire diff, making them much easier to read and making it harder to miss comments (#1334577)

We threw in a couple of bonus features here as well.  You can now see side-by-side diffs for merge proposals as well as unified diffs (#917502): there’s a link at the top of each preview diff allowing you to switch it to side-by-side mode (we may add a user preference for this later).  And resubmitting a merge proposal now automatically preserves the commit message (#676769).

Package build infrastructure

When we first added support for building debug symbol packages (ddebs) in Ubuntu, we handled the build and publication side of things with a temporary hack involving a custom hack to our sbuild fork that stashed the ddebs on the builder for a while, and a job that periodically fetched them to ddebs.ubuntu.com. As is the way of temporary hacks, it proceeded to stay that way for eight and a half years. It mostly worked, but was rather fragile, contributed to tying us to an old sbuild, wasn’t very extensible to PPAs, and couldn’t work with virtualised builders where all the builder’s state is reset before every build.

A couple of years ago we put together the Launchpad changes required to store ddebs straightforwardly in Launchpad instead, but weren’t able to deploy this at the time due to needing better librarian infrastructure. We finally got all this cleared out in April and deployed the new ddeb publishing mechanism. The main user-visible effect of this is that ddebs.ubuntu.com isn’t liable to lose packages any more, and ddebs are also available from Launchpad build pages.

Having done this, we were able to upgrade from our 2004-era fork of sbuild to a modern version, which fixed a number of long-standing bugs.

All architectures are now optional for PPAs (#1244868), so, for example, a PPA that only needs to support armhf can save builder cycles by disabling amd64 and i386. At the moment this can only be done by Launchpad administrators, so ask a question on Launchpad if you think this would be worth doing for any of your PPAs.

We rewrote the way that we install build-dependencies for source package recipes. This allows recipes to handle the “:native” qualifier in build-dependencies which is used in some places as part of supporting cross-building.

The next step in this modernisation programme will be to move all builds into ScalingStack and decommission the corresponding bare-metal builders, which we’ll be starting soon and working on an architecture-by-architecture basis as that cloud gains support for them. In the short term, this will give us better build capacity on the migrated architectures; in the longer term, it will let us support PPA builds on more architectures, and erase the cumbersome distinction between “non-virtualised” and “virtualised” builders. More news on this as it happens.

Performance

We upgraded to significantly more modern database servers earlier this year, which meant that a lot of hitherto difficult timeout problems suddenly either disappeared or became tractable. There are still some bad queries and a few mysterious problems, but as a general trend our timeout rates are very significantly down from where they were six months ago. We have some more general plans in this area, and will continue to spot-fix bad pages as they show up and as time permits.

Epilogue

We have lots more to do across the whole application, but still have a rather limited number of developers. If any of this kind of thing sounds interesting and you’d like to help, you can!

May 1, 2015
Git code hosting beta

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

  • push Git repositories to Launchpad over SSH
  • clone repositories over git://, SSH, or HTTPS
  • see summary information on repositories and the branches they contain in the Launchpad web UI
  • follow links from the Launchpad web UI to a full-featured code browser (cgit)
  • push and clone private repositories, if you have a commercial subscription to Launchpad
  • propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

  • useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
  • RSS feeds
  • mirroring
  • webhooks
  • an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Helping out

This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

October 19, 2014

Curtis Hovey
sinzui
Sinzui » Sinzui
Reverting Core Storage Physical Volumes

I after upgrading OS X to Yosemite, rEFInd did not start. Using the Option key while rebooting, I selected my Ubuntu Trusty partition, which booted with a warning that the MacintoshHD couldn’t be mounted.

During the upgrade, the HFS+ partition where OS X resides was converted to a logical volume of Apple Core Storage. I believe this is to support encryption, but I declined the option during the upgrade. The Disk Util app in OS X will not show this change, but the diskutil command will. The fix is trivial from the OS X terminal.

The diskutil list command will reveal which partition was convert to a logical volume on the physical disk.

diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         175.6 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
   4:       Microsoft Basic Data                         70.3 GB    disk0s4
   5:                 Linux Swap                         4.2 GB     disk0s5
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *175.3 GB   disk1
                                 Logical Volume on disk0s2
                                 41AF7850-5594-4959-A135-582BF79FDE46
                                 Unencrypted

Your disk partitions for dual booting Ubuntu and OS X may be different. I know there was no logical volume before I started the upgrade. I restored the APPLE_HFS partition to the real disk0s2 with the revert option

diskutil corestorage revert /dev/disk1

When I rebooted, rEFInd presented me with choice of OS X and Ubuntu. Ubuntu mounted with MacintoshHD when I signed in. All fixed.

June 12, 2014
Migrating juju to HP Cloud’s Horizon

HP Cloud retired its old regions this month. If you use juju on HP Cloud and your account was created before this year, you may need to update your juju config and possibly complete your project setup in Horizon.

There are two changes you must make to each HP Cloud environment listed in environments.yaml. Set the region option to “region-a.geo-1″ or “region-b.geo-1″, which are US West and US East respectively. The new regions do not automatically assign public IPs. The use-floating-ip option must be set to “true” to tell juju to request a public address.

use-floating-ip: true
region: region-a.geo-1

Note that juju does not yet release the public address when you destroy the machine, You need to use the nova command line tools or the Horizon console > Project > Compute > Access & Security > Floating IPs to release them. We will release a fix for “Floating IPs are not recycled in OpenStack Havana” soon.

If you haven’t used these regions before, you needs to register your public SSH keys and create a default network for each one you use. Visit Horizon console > Project > Compute > Access & Security > Key Pairs. Most people migrating just copied their public keys to the new region. Visit Horizon console > Project > Networks > Network and verify or create a default network, something like 10.0.0.0/24.

June 10, 2014
Building trans-cloud environments with juju

The Juju QA team uses juju to build the CI services that test juju. We have built 3 CIs in fact, because juju makes it easy to move services to the cloud of our choice. Our choice is driven by which cloud provides the best combination of resources, but CI has grown beyond what one cloud can provide. We want CI in all clouds to test with the resources they provide. While juju doesn’t support cross environment relations, that hasn’t stopped the QA team from building an hybrid environment that straddles two clouds and a private network.

The steps to provision a machine on another network are codified in our add-remote-machine.bash script. The Juju QA team uses this script to add physical machines, machines in kvm, and machines launched in other clouds into our Juju environment. Once a machine is registered with a juju environment, any charmed service can be deployed to it. Adding a machine to a open network is painless.

add-remote-machine.bash juju-ci3 my-keys/juju_ci_rsa 10.10.10.3
    |                          |          |              |
  script                     env-name   ssh-key   private-ip

Adding machines on restricted networks may require firewall egress changes.

Provisioning the best resources for services

Juju provides two essential devops features:

  • A cloud-agnostic way to provision machines. Juju supports Azure, EC2, Joyent, MAAS, and OpenStack-based clouds like HP Cloud.
  • A trivial means to deploy configured services to machines. I can put Jenkins slaves and web applications into production in minutes.

The mechanism to deploy services is so valuable, that juju provides a command to add existing machines to the environment. Existing machines often have special resources not provided by a simple cloud image. Using placement, I can deploy one or more services to each machine. For example, to add a machine that I want Jenkins to use, I would issue these commands to register machine number 2 in my environment, then deploy a jenkins slave to it, and finally configure the existing jenkins master to send work to it:

juju add-machine ssh:ubuntu@10.10.10.3
juju deploy --to 2 jenkins-slave ppc64-slave
juju add-relation jenkins-master ppc64-slave

Provisioning remote machines

The problem is that juju wants a private address when provisioning a machine. The solution to building hybrid environments composed of machines from other networks was understanding what network resources juju requires during and after provisioning. First, there are three parts to juju:

  • The juju client (juju command line or juju-gui) that issues commands
  • The juju state-server that manages the machines and services in the environment
  • The many juju agents (1 for each machine and service) that asks the state-server for tasks

During the act of provisioning (bootstrap or add-machine) the client acts as a bridge between the state-server and the remote machine. The private address is thus private to the host that the client is running on! Once the agent is running on the remote machine, it will talk to the state-server using the state-server’s private DNS name. The state-server doesn’t really know where the remote machine is, and it cannot access it.

Since I have ssh access to all the machines via their private address, I can add them to the juju environment. Even when I only know a machine’s public address, I can add an ssh rule that maps a random private address to the public host. As for helping the agent talk to the state-server, a single addition to the agent’s /etc/hosts is required to map the state-server’s public IP address to the private DNS name.

The script checks the required connectivity. It tries to verify common addresses that juju will use, such as the location of the environment’s private container, the server images, and juju tools. It is not authoritative. It will ask me to verify some connections that are not explicitly defined in the environment. It doesn’t know the network requirements of the services that will be deployed to the machine. This isn’t an issue for machines on open networks. Several of the Juju CI machines are on restricted networks, and I asked the IS department to allow egress to the required addresses and ports. We may improve the script’s verification support as we extend the environment to HP Cloud and Joyent.

As for how Juju makes CI easier, that is the subject or a future post.

April 21, 2014
Working with Juju, Ubuntu 12.04 Precise, and 14.04 Trusty

Juju 1.18.x is the first Juju that is aware of two Ubuntu LTS releases.
In the past, when bootstrapping the state-server or deploying a charm,
Juju selected Ubuntu 12.04 Precise when the serie was not specified. If
you care about the version of Ubuntu used, then you want to specify the
series

You can specify the series of the state-server and set the default charm
series by adding “default-series” to environments.yaml. Setting
“default-series” is the only way to specify the series of the
state-server (bootstrap node). For example, if your local machine is
Ubuntu 14.04 trusty, but your organisation only supports Ubuntu 12.04
Precise in the cloud, you can add this to your environment in
environments.yaml:

default-series: precise

There are many ways to specify the series to deploy with a charm. In
most cases you don’t need to. When you don’t specify a series, Juju
checks the environment’s “default-series”, and if that isn’t set, Juju
asks the charm store to select the best series to deploy with the charm.
The series of the state-server does not restrict the series of the
charm. You can use the best series for a charm when deploying a service.

When working with local charms, Juju cannot fall back to the charm
store, it falls back to the environment’s “default-series”. You must
specify the series in the environment or when deploying the charm. If
your environment is running, you can add “default-series” like so:

juju set-environment default-series=precise

These commands choose Ubuntu 12.04 Precise when “default-series” is set to “precise” in the environment:

 juju deploy cs:precise/mysql
 juju deploy precise/mysql
 juju deploy mysql
 juju deploy local:precise/mysql
 juju deploy loca:mysql

November 16, 2013
Restoring network to lxc and juju local-provider

I have experienced two cases where lxc containers stop working or new containers never work because they cannot join the network. My LTS container started, but without network, there were a lot of things I couldn’t do in it. In the case of Juju, I couldn’t deploy new local services. The unit agent status was stuck in PENDING.

You can verify the network is broken by fancy listing the lxc containers you started either by lxc-start or juju deploy:

sudo lxc-ls --fancy

NAME        STATE   IPV4 IPV6 AUTOSTART 
-------------------------------------------
<container> RUNNING -    -    NO

This shows that the running container doesn’t have IPV4 or IPV6 networks.

For each existing container that needs fixing, you need to install the new dhcp packages, but without a network, you cannot do it from a running container. Instead chroot can be used to update the container’s root file system.

cd /var/lib/lxc/<container>/rootfs/
sudo chroot ./
sudo apt-get update
apt-get install isc-dhcp-common isc-dhcp-client

To ensure that new containers work, you need to clear the lxc cache. The lxc image cache was built when you created or deployed your first container. The images there are probably more than a few months old.

sudo ls -lh /var/cache/lxc/precise/
sudo ls -lh /var/cache/lxc/cloud-precise/
sudo rm -r /var/cache/lxc/*

Lxc will get the new images the next time it needs to create a container. The first call will be slower to complete since it will build a fresh cache.

October 28, 2013
Restoring grub after OS X Mavericks

I had a scary moment after I updated my dual boot MacBookAir to Mavericks. Refit started Ubuntu, but grub was showing me the rescue prompt. This was a different misadventure from my upgrade to Mountain Lion; the partitions were there, but grub was lost.

I use refit to manage dual boot. I remembered from past experience that I needed to use the partition tool from the refit boot screen to sync the GPT and EFI tables. The task was done in seconds.

I knew Ubuntu was on the 4th partition (suggested by the GPT output from the sync), but to be sure I listed the partitions, then listed the 4th one.

ls
ls (hd0,gpt4)/boot

and I saw the boot images and a grub dir. I then assumed, all my data was in place, but the MBR was lost during the Mavericks install. To start Ubuntu, I typed:

set root=(hd0,gpt4)
set prefix=(hd0,gpt4)/boot/grub
insmod normal
normal

Ubuntu started up as normal. The grub screen was identical to my last boot. I was confident grub’s configuration was fine, I just needed to restore grub to the MBR. I opened a terminal after logging in and typed:

sudo grub-install --force /dev/sda

With hesitation, I rebooted. All was back to normal.

September 2, 2013

Launchpad blog
lp-blog
Launchpad blog
Launchpad build farm improvements

We’ve made a number of improvements to the Launchpad build farm in the last month, with the aim of improving its performance and robustness.  This sort of work is usually invisible to users except when something goes wrong, so we thought it would be worth taking some time to give you a summary.  Some of this work was on the Launchpad software itself, while some was on the launchpad.net hardware.

(To understand some of the rest of this post, it’s useful to be aware of the distinction between virtualised and devirtualised builders in Launchpad.  Virtualised builders are used for most PPAs: they build untrusted code in a Xen guest which is initialised from scratch at the start of each build, and are only available for i386, amd64, and a small number of ARM builds by way of user-mode QEMU.  Devirtualised builders run on ordinary hardware with less strict containment, and are used for Ubuntu distribution builds and a few specialised PPAs.)

ARM builders have been a headache for some time.  For our devirtualised builders, we were using a farm of PandaBoards, having previously used BeagleBoards and Babbage boards.  These largely did the job, but they’re really a development board rather than server-class hardware, and it showed in places: disk performance wasn’t up to our needs and we saw build failures due to data corruption much more frequently than we were comfortable with.  We recently installed a cluster of Calxeda Highbank nodes, which have been performing much more reliably.

It has long been possible to cancel builds on virtualised builders: this is easy because we can just reset the guest.  However, it was never possible to cancel builds on devirtualised builders: killing the top-level build process isn’t sufficient for builds that are stuck in various creative ways, and you need to make sure to go round and repeatedly kill all processes in the build chroot until they’ve all gone away.  We’ve now hooked this up properly, and it is possible for build daemon maintainers to cancel builds on devirtualised builders without operator assistance, which should eliminate situations where we need urgent builds to jump the queue but can’t because all builders are occupied by long-running builds.  (People with upload privileges can currently cancel builds too, which is intended mainly to allow cancelling your own builds; please don’t abuse this or we may need to tighten up the permissions.)  As a bonus, cancelling a build no longer loses the build log.

Finally, we have been putting quite a bit of work into build farm reliability.  A few problems have led to excessively long queues on virtual builders:

  • Builders hung for some time when they should have timed out, due to a recent change in su; this is now fixed in the affected Ubuntu series.
  • Xen guests often fail to restore for one reason or another, and when this happened builders would fail in ways that required an operator to fix.  We had been dealing with this by having our operators do semi-automatic builder fixing runs a few times a day, but in recent months the frequency of failures has been difficult to keep up with in this way, especially at the weekend.  Some of this is probably related to our current use of a rather old version of Xen, but the builder management code in Launchpad could also handle this much better by trying to reset the guest again in the same way that we do at the start of each build.  As of this morning’s code deployment, we now do this, and the build farm seems to be holding up much more robustly.

This should make things better for everyone, but we aren’t planning to stop here.  We’re intending to convert the virtual builders to an OpenStack deployment, which should allow us to scale them much more flexibly.  We plan to take advantage of more reliable build cancellation to automatically cancel in-progress builds that have been superseded by new source uploads, so that we don’t spend resources on builds that will be rejected on upload.  And we plan to move Ubuntu live file system building into Launchpad so that we can consolidate those two build farms and make better use of our available hardware.

June 25, 2013

Curtis Hovey
sinzui
Sinzui » Sinzui
Managing Juju Charm Versions

It is difficult for Juju charm authors to support forked charms. dev-ops often don’t realise they are forking a charm when they add files that they want the charm to deploy. We cannot assume that the deployed charm’s revision (or the version control revision number) will ever match what the author released. I use Bazaar tags to mark the versions I support and help identify the true version of a deployed charm.

Juju uses the charm’s “revision” file to store the version. The number in this file must be incremented to deploy additions or changes to the charm. Upgrading a deployed charm requires a higher revision number. Organisations commonly create a local charm repo (managed by a version control system like Bazaar) to freeze the versions they deploy. These known version are stable; this is a best practice when working with charms that change often. The local branch will get commits that are not in the branch published at Launchpad. The revision numbers are meaningless between the two branches.

I tag my charm branches with the charm revision number. When a dev-op forks my charm to add it to the local charm repo, we can compare branch tags. For example, I tagged lp:~charming-devs/charms/precise/elasticsearch/trunk r39 as elasticsearch-29 to match the number in the revision file. I can ask the dev-op which tags the local branch has. I can quickly identify missing releases. I can ask for a diff from the tag I last released to help diagnose problems with the changes.

For example, several issues were reported about my elasticsearch charm. I had addressed the problems months before. I suggested the dev-ops upgrade to the latest version. Since their version numbers were higher than my versions, they did not upgrade. Tags are more precise. I can suggest the dev-ops merge “-r tag:elasticsearch-29″.

Another case, charms may need to change often if the application is deploys is under active development. A recent deploy of charmworld failed; the charm upgrade errored. The dev-op and I couldn’t reconcile the version numbers between what was deployed and what I released. We had to read a diff of the charm’s directories and files to discover the deployed charm was missing several releases–the charm was incompatible with the deployed code. If I had tagged my release, we could have identified the issue in minutes.

June 6, 2013
Closing milestone bugs using Launchpad’s API

A few years ago I wrote a contrib script for Launchpad’s launchpadlib called ‘close-my-bugs.py’ which attempted to close (aka mark them ‘fix released’) all of your bugs in a project that were targeted to a particular milestone.

For various reasons it grew out of date and when I needed to use it recently, it didn’t work!  Long story short, I just fixed it up and added a couple of new features:

  • You can optionally close just your own bugs, or all the bugs in the milestone
  • You can search for bugtasks targeted against a series in your project (these are not normally picked up when searching in a project’s milestone)

You can grab the code here:

bzr branch lp:launchpadlib

contrib/close-my-bugs.py

April 24, 2013
Error handling in Go

There’s been a debate raging in some corners of the internet lately about how superior Go‘s error handling is to other languages.  I am going to address some of the points made, here:

Claim 1: It’s impossible to ignore errors in Go, they are “in your face”

This is patently false.  Take this example:

fmt.Println("Hello world")

Pretty innocuous wouldn’t you say?  Well let’s take a look at the language documentation for fmt.Println:

// Println formats using the default formats for its operands and writes to standard output.
// Spaces are always added between operands and a newline is appended.
// It returns the number of bytes written and any write error encountered.
func Println(a ...interface{}) (n int, err error)

So Println can return an error!  Where did we check it?  Well, we didn’t.  Any other claims that it’s OK to ignore it in this case further strengthen my argument.

Some will say that it’s a deliberate choice to ignore the error and I deserve all I get. Well, was it? I didn’t even know that Println returned an error until I looked at the documentation (and who is going to do that for Println?). And that’s the point, if I need to look at the documentation to see that it can return an error, then if I am using a language that raises exceptions I will have also seen its documentation about how it deals with errors.

You could even argue that an exception is superior in this case.  With Go, the code will march on regardless, oblivious to the fact that Println failed.  With exceptions, it’ll fail and show you exactly where it failed.

The language will error at compile time if you try to ignore an error returned as a second value and you only take the first.  But this is trivially bypassed by assigning it to _, which when reading code is easily missed compared to the exception style of “catching then dropping”, because Go itself encourages this style of assigning to _ with its own range statement as a deliberate way of ignoring things that the language is trying to force you to see.

So really in both cases, ignoring the error doesn’t really stand out as wrong.

Here’s a concrete example in Go I was recently shown:

w := bufio.NewWriter(os.Stdout)
for _, name := range ListAll(conf) {
    fmt.Fprintln(w, name)
}
w.Flush()
return

As you can see, the caller completely forgot to check the error returned from Fprintln and Flush and there would be no compiler warning about it.

Claim 2: Exceptions teach developers to not care about errors

Citing an example where someone didn’t catch an exception and the code consequently blew up is really not a good example of this claim.  It’s a bug, for sure and you get a full traceback of your error in the resulting exception, which is handy.  You go away and fix it quickly based on that info.

If I am in the same situation with Go and I ignore a returned error from a function, at some point (which is likely to be nowhere near the place where the error occurred) my code will blow up.  I’ll have to run up the debugger to try and find out where it really occurred though.

Because unused variables in Go are a compile-time error, it’s actively discouraging you from assigning the result of the function to a variable (or you can deal with it, of course).  For anyone who’s not read the full documentation for a function call or missed its return value (we’re all human) as I said above – you’re not even going to notice that you missed it.

Based on this, I can see no difference at all that suggests one way or the other teaches developers to not care about errors.  Developers do care about errors, really, but bugs creep in however careful you are.  And when they do, I’d rather have a decent indication of where the bug is.

Other parts of error handling that I dislike

When you look at the average Go program, you will see a lot of this:

if err != nil {
  return nil, err

This is the recommended way of error handling in in Go.  But this is not error handling, it’s error propagation.  In nearly all languages there will arise situations where in well-factored code you have a low-level error that you need to pass right back up to the entry point for the caller. That means you need this error propagation code in every single place where you check for errors.  There’s no syntactic sugar, just the same three lines everywhere.

For me, this vastly decreases the readability of the code. This is where exceptions excel because inside my own library I can factor the bejeesus out of it into many small functions and if I need to return an error, I just catch a lower-level exception in the top-level function and return something else.  You can do this in Go with a panic(), but it seems to be discouraged.  Panic() feels almost exactly like using exceptions, only the syntax is worse. If Go’s style is to encourage people to handle errors like this, it needs the sugar.

Conclusion

Many people might think that I completely hate Go’s error handling from this post.  That’s not strictly true – I don’t hate it, I just think it can be improved.  I challenge assumptions that I see which state that Go’s error handling is superior in some way, when as far as I can see it’s not that different from other languages in terms of usefulness.

Go is clearly in its infancy.  Most languages will have started out with youthful enthusiasm and realised that some change was needed.  These languages are the successful ones where developers enjoy coding in it and feel productive.  I hope that Go embraces change as it matures and attracts more developers.

I welcome comments on this post – unlike some people I won’t censor them or delete ones I can’t argue with (unless they are outright abusive and use foul language, this is a family blog!).


December 4, 2012

Launchpad blog
lp-blog
Launchpad blog
Private Projects and Private Blueprints leave beta

Today, the Private Projects and Private Blueprints features on Launchpad are leaving beta. These features are available now for use by all Launchpad users. Private Blueprints was started as part of the Private Projects work, with the end goal in mind of truly private projects on Launchpad. Private Projects was described in its beta announcement like this:

When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.

When you create a proprietary or embargoed project on Launchpad, all of the sharing policies for your project will be set correctly for you. This means that if you start your project as a proprietary project, your branches, bugs, and blueprints will be created proprietary by default. Answers and translations are not available for private projects.

A commercial subscription is required to use private projects, but any user who creates a proprietary or embargoed project on Launchpad will receive a 30 day trial commercial subscription. Launchpad users with existing commercial subscriptions can convert a public project to proprietary or embargoed by changing the information type in the project’s settings. You may have some work to do on your project before you can transition to a private information type — for example, disable answers if you have that app enabled for your project — but Launchpad will block the change and tell you what needs to happen before you can switch to a private information type.

Users should be aware, though, that if your project has been listed on Launchpad publicly until now, then search engines know of its existence already. If you want a proprietary project that no one can learn of its name, you should create a new project on Launchpad. Transitioning a public project to private allows you to keep your series and milestones private going forward, but users may have already been able to discover the existence of the project since it was public already.

We are happy to make truly private projects available for all users on Launchpad. If you run into any issues, please file a bug against Launchpad or ask for help in #launchpad on Freenode.

November 20, 2012
How Novacut uses Launchpad

Launchpad has been a key tool used in developing Novacut. I use Launchpad for code hosting, bug tracking, daily builds, and more. For almost two years I’ve been doing monthly stable releases on Launchpad, and Novacut now spans six separate Launchpad projects. To say the least, I’ve learned a lot about Launchpad in the process.

I don’t think Novacut could be where it is today without Launchpad, so I want to pass on some of what I’ve learned the past two years. Here are my five essential Launchpad best practices:

1. Daily Builds

I’m always very thankful that early on Paul Hummer took the time to school me on using Source Package Recipes to do daily builds. This Launchpad service gives you automated package builds across multiple architectures, and multiple Ubuntu releases.

I don’t know how to emphasize this enough, but seriously, you need daily builds. As a point of reference, daily builds are the 3rd item in the famed Joel Test.

These builds are triggered simply by making commits to the appropriate bzr branch on Launchpad (usually your trunk branch). You’ll automatically get up to one build per 24-hour period, and you can manually trigger additional builds when needed.

You can include your debian/ packaging directory in your project source tree, or you can keep debian/ in a separate bzr branch. For the Novacut components, I’ve found it most helpful to keep debian/ in the source trees because it’s handy to be able to land a code change and its corresponding packaging change in a single merge. This works for us because we currently can use the exact same debian/ for all the Ubuntu versions we support. If that’s not true for your project, you’ll need multiple debian/ branches.

For reference, here’s the Novacut Source Package Recipe.

2. Unit Tests

You should run your unit tests during your package builds, and you should fail the build when any unit test fails. This is particularly important for daily builds, because this will prevent a package with broken unit tests from reaching your daily PPA.

The Launchpad build servers are strict and unforgiving environments, which is a good thing when it comes to unit tests. The build servers are also probably quite different from your local development environment. On countless occasions our daily builds have caught failures that only occur on i386 (my workstation is amd64), or only occur on an Ubuntu release other than the one I’m running, etc.

To run your unit tests during the package build, you’ll need to modify your debian/rules file as appropriate. If you’re using debhelper, add an override_dh_auto_test target.

You might also need to add additional packages to the Build-Depends section of your debian/control file, packages that are needed by the unit tests but are otherwise not needed by the build itself.

For reference, here’s the debian/rules file used to run the Dmedia unit tests (which is also a handy Python3 example).

3. Track Ubuntu+1

When a new Ubuntu version opens up for development, I immediately start doing daily builds on the development version, even though I don’t typically upgrade my own computers till around 4 months into the cycle.

I use daily builds on the development release as an early warning system. With no extra effort on my part, these builds give me a heads-up about code or packaging changes that might be needed to make Novacut work well on the next Ubuntu release.

To enable daily builds on the next Ubuntu version, just go to your Source Package Recipe, click on “Distribution series”, and check the box for the newest series. Now you’ll have daily builds on the newest Ubuntu version, in addition to all the versions you were already building for.

For example, I’m currently in the process of enabling daily builds for Raring, as you can see in the Microfiber Source Package Recipe. And I did indeed encounter a build failure on Raring, seemingly caused by a debhelper issue.

For the first month or so in a cycle, I don’t tend to worry much about build failures on the development version. But after the dust has settled a bit, I make sure to keep the builds in working order, and I even do monthly stable releases for the Ubuntu development version. Again, I do all this pro-actively even before I personally start running the newest Ubuntu version.

4. PPAs & Users

Whenever someone asks me why I use Launchpad instead of github, my short answer is always, “PPAs and users”.

Source Package Recipes give you much more than just a build, they give you daily packages that are easily consumable by your testing community and early adopters. This tight feedback loop prevents you from running too far ahead without getting a good reality check from your target users.

Keep in mind that for some products, the early adopters willing to install from a PPA might not be all that representative of your target user. So when it comes to making design decisions, you might need to politely ignore certain feedback from some of these early adopters. In my experience, this wont cause any hard feelings as long as you have clearly communicated who your target user is, and why.

For reference, you might look at the way we’ve defined the Novacut target user.

I recommend creating PPA names that are well-branded and easy to remember. First, create a Launchpad team with the same name as your product. In our case, we have a ~novacut team. Second, I recommend creating a daily and a stable PPA owned by the same team. In our case, that gives us two easy to remember PPAs:

Although none of our target users (professional video editors) currently use Ubuntu to do their job, I’ve been surprised by how many follow Novacut’s development via our stable PPA, and even our daily PPA. This has helped keep us on track, and has helped us build customer loyalty even before we have a finished product.

For me personally, this daily user engagement also makes the design and development process more enjoyable. It’s hard to empathize with an abstract persona; it’s easier to solve specific problems for specific people.

5. Use Apport

Till recently I didn’t realize that you can use Apport for automated crash reporting in unofficial packages delivered through a PPA.

We haven’t had Apport integration for that long, but it’s already provided us with dozens of highly valuable crash reports. Almost immediately some hardware specific issues came to light and were fixed, convincing me that a key benefit of Apport is knowing how your app might misbehave on a larger, more variable pool of hardware.

Apport also helped some rare bugs come to light. I thought Dmedia was basically crash-free, but those one-in-a-thousand bugs pop out quickly when thousands of people are running it. Most of these bugs would have eventually been found by one of our core devs, but the quicker a bug is discovered, the quicker and easier the bug is to fix.

For more info, check out this blog post and this screencast, where I covered our Apport integration in detail.

And for reference, see the merge proposal that added Apport integration in Novacut.

A big thank you to Jason DeRose for sharing how his project uses Launchpad on a daily basis.

 

November 9, 2012

Deryck Hodge
deryck
Considering Tumblr

I've grown tired of self hosting this site. I think it affects how much I blog or write online because I wrote this site myself, it's a bit heavy weight compared to more recent tools, and I just don't have the time or motivation to work on the site anymore. I just want to blog, link to interesting stuff, post pictures, etc.

Therefore, I'm considering moving this site to Tumblr. Now's your chance to weigh in and tell me I'm crazy. Or not.

What say ye all?

November 6, 2012

Launchpad blog
lp-blog
Launchpad blog
The information sharing feature is complete

Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

The Disclosure features

Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.

There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing.   We did not consider Sharing, or any of the required features complete until we fixed all the bugs.

Disclosure facts

  • Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
  • Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
  • 149 related bugs were identified in Launchpad.
  • Work started in June 2011 by the Purple squad.
  • Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
  • About 45,000 lines code were added to support the features.
  • About 15% of the lines were for missing JavaScript test coverage.
  • More that 700 bugs were fixed in total.
  • About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
  • About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.

Lessons learned

  • Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
  • Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
    • Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.
    • Privacy was added on top of branches, and it failed to scale to 1000’s of branches.
    • Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.
  • Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
  • Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
  • Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.

October 25, 2012
Private projects for beta testers

If you are part of Launchpad’s beta testers team, you can now start trialing private projects on Launchpad. The private projects feature builds on the great sharing work that Launchpad’s Purple Squad has done, allowing Launchpad users to create true private projects now. A commercial subscription is required to use private projects, but any user who creates a private project on Launchpad will receive a 30 day trial commercial subscription.

When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.

Be warned, this is a large change to Launchpad and there are certainly bugs in our handling of privacy. You can check out our list of known issues, if you’d like. We, as the Launchpad Orange Squad, are committed to fixing all of those before we leave beta. So don’t worry, we’re still actively working on this feature. We did, however, want users to begin using this feature to get early feedback on the work. Don’t trial your super secret project with this feature just yet, but if you have something safe to try out private projects, now is a good time for beta testers to get going with the feature.

Enjoy private projects on Launchpad now, beta testers! And please file any bugs you find.

October 23, 2012
Launchpad Workshop at UDS-R

After the success of the Launchpad clinics at the last UDS-Q we’ve decided to run some more! This time removed the sterile name of clinic and called them workshops.

If you want to get involved, scratch that itch, learn how to fix that irksome bug that has been bugging you’re not alone. Everyone probably has at least one that they’d like to see fixed.  The problem is now knowing how to fix them or maybe they don’t know how to set up the Launchpad development  environment, well lucky for you we have a lot of Launchpad developers at UDS-R and we’d like to help you help get bugs fixed!

The idea being if you have a bug you would like to fix, or pointed in the right direction  that we’ll be there to help you get on the road  to offer advice on every step of the Launchpad development process from Lines of code, to branch reviews to getting things done. We’ll have EC2 instances ready for you to develop on, so if you haven’t already gone through the process of setting up local Launchpad development on your machine, you don’t need to worry.

I have created a wiki page on which you should register if you’re going to be attending either of the clinics. Just list your name and the ID of the bug(s) you want to work on on that page. We’ll check the bugs out and get in touch with you if we think they’re too big to work on in the clinics – in which case we’ll try and work with you to get them fixed over a longer period. We’ve added the event to summit schedule, for Tuesday and Thursday of UDS so why not sign up and come along!

If you’ve never contributed before, Graham Binns has written a useful guide to contributing to  Launchpad.  He has also done up a screencast on fixing a bug in Launchpad.