Opening Ubuntu Up To the World

Recently I have been working with David Planella and Michael Hall on my team around a new specification for empowering app developers to deliver their content in Ubuntu. This post provides some background information around this work and the problem it seeks to solve.

Like many of you, I am hugely proud of the progress we have made with Ubuntu over the years. We have worked together to create a simple, powerful experience underlined with the foundation of our core Ubuntu values of creating a free platform, available to all, in your language, irrespective of (dis)ability.

While our platform has been growing and maturing, in recent years we have been presented with a new challenge that we need to solve: making it simple for content creators to deliver their content in Ubuntu.

Ubuntu is not the only thing that is maturing. Mother nature needs a helping hand to keep this solid slab o’ Bacon in it’s prime.

A Little History

When Ubuntu was started the competitive landscape was very different. We focused on being the best Linux distribution we could and making Ubuntu as powerful, reliable, and flexible as possible. Back then our primary competition was Microsoft with their Windows platform, but we could compete there by making our own, better platform. Ubuntu was faster, more secure, had more choice and other benefits. With these benefits we started to grow.

The competitive landscape today is very different. No longer are we content with just being the best Linux distribution, but we are going head-to-head with Apple, Microsoft, and Google with IOS, Windows, and Android respectively. It is not enough to just make a better, more stable, and more reliable platform these days; a reasonable assertion of success is giving users the content that they want.

To a large extent this boils down to applications. People don’t just choose their devices and platforms on the competence of the platform, but also on the apps (often specific app brands) that they want to do their work, manage their lives, and relax.

And here lies the challenge.

A few years ago we identified that we have a problem in Ubuntu with the complexity and expectations of how app developers get their apps into the Ubuntu Software Center. Please note: in this post I am referring to Free Software and Open Source apps; commercial app developers can submit their applications and have them reviewed by Canonical and this works pretty efficiently. The challenge here (ironically), is with Free and Open apps.

Now, when I say app developers here, I am not just referring to the upstreams that we know and love who already have a close relationship with Linux distributions such as Ubuntu. I am instead talking about the long tail of app developers who frankly don’t really care about the Operating System but just want to deliver their applications on it. These folks are typically excited by Ubuntu and the opportunity of Ubuntu, but they don’t want to be involved in the nitty-gritty of how Ubuntu is built; they care about Ubuntu merely as a platform to get their apps out to more users. There are literally thousands of application developers out there like this and I believe that for Ubuntu to be successful, we need to engage and empower these users.

No longer can we merely fall back on Hungry Hippos to engage and empower our users.

When we discussed this a few years ago we identified the following problems with our current processes for getting apps into the platform:

  1. Application developers could not deliver their apps to the stable release of Ubuntu and could only get them into the development release (in preparation for the next stable release). Of course, there was the backports team that serves getting apps onto the stable release…but this leads us to the next point…
  2. There was a particularly high bar to deliver your app in Ubuntu. You either had to be an Ubuntu Developer, which is a role designed for Operating System integrators (which is not of interest to app developers) or you needed to know an Ubuntu developer who could upload on your behalf (which does not scale to most app devs…we don’t have that many Ubuntu developers :-) ).

These approach was frankly, not all that surprising: Ubuntu inherited much of its process and workflow from how Debian works, and this process and workflow is designed for Operating System integrators and people who fundamentally care about creating and building an Operating System as opposed to content creators whose priority is their app and not the platform. This is not to say that app devs don’t want to harness the platform and ensure that their apps runs well, but their priority is their app and the connection points to the platform and not the platform itself.

To resolve this we put together a new process in which app developers could submit their applications to the Application Review Board (ARB), which is a community team that performs a security and code review and assesses the suitability of the app going into the archive (a different part of the archive called extras). Importantly, these applications, when approved by the ARB would be delivered to stable releases, thus solving the first bullet above.

As the years passed it became clear that despite the admirable efforts of many members of the ARB, the process was simply not set up for success. The ARB struggled to keep on top of the queue of applications that came in, and very few applications successfully got through the process. From the app developers perspective this was frustrating; their application entered the queue and typically took months to get through, if at all. Again, kudos to the ARB for their efforts, but the source of the problems was not the members of the board but the process: requiring a security and code review for every version of every app submitted was never going to scale effectively.

I wish this would “scale more effectively” too.

A good example of this was the recent Ubuntu App Showdown. We had 133 applications submitted to the contest and the ARB buckled under the load of the reviews. If we face this problem with the relatively small number of apps coming in currently, as Ubuntu grows as a consumer-grade product, we will face this even more. Put simply: Ubuntu is not currently optimized for the needs of free app developers in delivering their apps on Ubuntu.

The problems with our current processes were also well known to app devs. A while back I kicked off a campaign with my team to encourage app developers to put a button on their websites that would provide a direct link to download the version of their software in the Ubuntu Software Center. A significant number of app devs were resistant to this as the versions in the Ubuntu Software Center were simply too old and their view was “why would I recommend my users download the old version in Ubuntu when I can instead ask them to install this PPA or download this Deb“.

In other words, free app developers are unable to deliver their recent releases to Ubuntu users via the Ubuntu Software Center. In my mind this means our current processes are failing them, and this will inhibit the opportunity of taking Ubuntu and Free Software to the masses.

A New Process

To be honest with you all, this issue has been frustrating me for a while. I see it in pretty blunt terms: if we don’t solve this issue of app developers being able to deliver their content to Ubuntu, Ubuntu is simply not going to be successful. If we don’t solve this problem our users won’t want to use Ubuntu as they can’t get the apps they want and app developers won’t want to deliver apps to Ubuntu as getting them into the Ubuntu Software Center is such a nightmare or impractical.

Of course, we could continue with the traditional processes I outlined earlier, but I believe this will always relegate us to an awesome Linux distribution as opposed to chasing the real opportunity of bringing a Free Software Operating System to the masses with the apps that people want and a fantastic opportunity to expose Open Source and Free Software apps and the hard work of their developers to more people than ever before. People talk about the chasm and how to get over it, I believe we need to solve this issue for us to get over it.

About three weeks ago I called a meeting with my team with the goal figuring out how to resolve this. Solving this problem properly was going to mean allowing app developers to upload new releases of their apps directly to Ubuntu safely and securely without requiring the manual reviews that caused the bottlenecks with the ARB. To achieve this outcome though there is a lot of work to be done in terms of application insulation and sand-boxing, tooling improvements and other things.

The security team hard at work sand-boxing.

I first started surveying various engineering managers and engineers in Canonical to see what work they were doing along these lines to get an idea of current resourcing (as Canonical will likely need to invest most in delivering this work), and then David Planella, Michael Hall and I spent a lot of time putting together the first cut of a proposed specification that would resolve this issue in Ubuntu. I am proud of the output of this work: the specification is crisp, detailed, includes clear design guidelines, and takes the reader through every layer of delivering such a vision. The specification takes into account current resourcing, and breaks the work down to the work item level that we can discuss at UDS. As part of this process we gathered a lot of input from people such as Jamie Strandboge, Steve Langesek, Michael Vogt, Matthew Paul Thomas, Allison Randall, and the the ARB. This feedback helped us to ensure that the specification is detailed, practical, and reflective of the needs of our users, while being safe and secure.

Big process discussions like this can often turn into unmanageable spidery mailing list threads that lose people when they hit a certain size. My goal here was to put together a really detailed first cut of a spec that we could first take to the ARB and then to ubuntu-devel and people could then discuss the specifics of the spec, debate changes, and otherwise get people together around a document and a specific approach. This could help focus the discussion on refining and improving that approach, as opposed to bike-shedding until everyone’s fingers fall off. You can read the on-going discussion here.

Like many of you, I really care about Ubuntu. I want to see us succeed, and I want to see us deliver the very goal we set out with in bringing the very best Free Software platform to the world. I think there is a tremendous amount of opportunity here, and easing how content creators such as app developers can get there content into Ubuntu will not only enrich the Ubuntu ecosystem, but provide more choice for our users and make us a more compelling platform when people choose which computer or device to use. I look forward to the discussion over the coming weeks and at UDS.

Thanks for reading!

  • bugger

    we can only succeed when Canonical focuses more on the Ubuntu Community by recognizing greatness and rewardingnthe countless hours of work contributors do for free. its also important to break down the walled garden Canonical is building around this OS.

  • Anonymous

    ??

  • Anonymous

    Hi! I intend to constructive criticism. Do not misunderstand my opinion. The App Showdown was wonderful and I really enjoyed all the process, but all could be better :)

    I participated in the Ubuntu App Showdown with PYEnglish app. One of my targets was to know how will be difficult to enter in the USC.

    Well, my experience was bad with the USC process. Why? Because my app entered after a lot of weeks and the more important to me: my app was in the “new apps” section a few hours only :( Then nobody can see it as I was waiting.

    Will have Steam the same problem too? I think Ubuntu must review those things in the future.

    Best regards! :)

  • Anonymous

    Yeah, though I am not a developer I see new applications are only visible in what’s new section for a while until they are replaced by newer software or more often by books and magazines. That’s why I proposed this section should be split into 2 on brainstorm.ubuntu.com In fact, if you have some idea in your mind I recommend you to share it on that web.

    @Jono Bacon – thank you and others in the team for improving Ubuntu. I like it and I enjoy every bit of it.

    Btw. What’s going on between Valve and Canonical? I am so excited about it.

  • Thomas Kluyver

    I look forward to seeing this. I’ve thought for some time that Linux distros need to draw a bigger distinction between ‘components’ – conservatively updated to avoid breaking things – and ‘applications’ – aggressively updated for users who expect the latest and greatest. It’s gratifying to see Ubuntu moving in that direction.

    Undoubtedly there will still be some bumps on the road, but nothing ventured, nothing gained.

  • http://blastfromthepast.se/ Tommy Brunn

    Kudos on the great work you and your team have been doing. I looked over the spec when it was released, and I believe it will greatly improve the process. My submission for the Ubuntu app showdown is still waiting for a review, so it’s pretty obvious that the process really needs to change.

  • David Planella

    That’s exactly the reason why we’re proposing an improved new process: we want to make it easy to app developers to submit their apps.

  • http://profiles.google.com/krnekhelesh Nekhelesh Ramananthan

    Jackpot! You have the nail on the head! I believe that one of the greatest strengths of Ubuntu is the USC. And if it is possible to make uploading apps easier and at the same time secure (sandboxing etc) it would be really awesome!

    At first I was a bit worried that app developers could only install their app at /opt, but looking at the spec I am glad to see that change.

    I am also hoping that this new Spec and the Quickly reboot go hand in hand. Kudos to make life for developers so much more easier than any other distro!!!!!!!

  • waspinator

    I am glad Ubuntu has finally seen the need for this. I am surprised that after seeing steam in 2003 and the apple and android app stores in 2008 that it took this long to start thinking about it. I hope we see a successful USC by 14.04.

    I also want to remind canonical that in order to attract developers you need to hhave great development tools. Quickly is a good start but there needs to be a great UI for it and all the boilerplate and best practices built in.

  • http://twitter.com/Sohaeb Suhaib Nabiel

    Finallyyyyyyy. No more feedback, suggestions and Talking ! ONLY working and trying to solve the problems :)

  • Gordintoronto

    How do you stop an author from putting a keylogger into version 2.1 of his app?

  • http://twitter.com/becket1 Bob K.

    When is Linux going to start having some prosumer/professional media creation software. Apple and Windows have the support of Adobe, Corel, and many other high end applications. Linux users don’t have the same level of tools at their disposal. How can Canonical attract developers of professional media/creation applications. Not everyone can afford a Mac!.

  • Anonymous

    Ubuntu was getting its rough edges sorted out; still doing that. As jono pointed out, the vision so far has been to become best linux distro. I believe that is achieved. It is time to look beyond that vision which they are doing. This is great.

    I don’t think it is too late to join in the app store game. Microsoft is still just getting warmed up. Don’t forget that USC is starting with truck load of apps.