Money. Stability?

Something struck me yesterday, while Firefox 3 was throwing a hissy fit. Is there some kind of correlation between the amount of money invested in a project or product and instability? I know this sounds like a bit of a negative question, and I don’t mean to rag on the Firefox folks, but it seems that as more money has been pouring into Mozilla Towers, the quality has been compromised. At first I thought this was just on Linux, but it seems to be on Windows too.

This is by no means an issue specific to Mozilla though – look at Microsoft with Vista. They filled the worlds largest boot full of money and poured it into the Vista machine and…bang! Instability. The more I think about it, the more examples occur to me.

Is it money directly? Or is it really that more money means more staff employed and instead the quality failure is due to management and coordination problems? Maybe it is more users that causes the problems – more users means more needs and requirements which maybe cloud the QA process? Alternatively, do the quality problems really exist, or are our expectations higher than they were? Do we expect more of our two examples, Firefox and Vista, than we did many moons ago?

I would love to hear your thoughts and experiences on this folks. I wonder what hints we can uncover.

  • http://aruiz.synaptia.net Alberto Ruiz

    Jono, do you have any sort of information to back that statement up? Which versions of Mozilla Firefox are you comparing? OTOH Vista is a huge chunk of code that cannot be tested in all the different combinations people is going to use it before it’s released, so I don’t really see the point here.

    The only think that you can state is that having more resources available doesn’t bring quality assurance out of the blue. But there are a lot of wealthy projects that are quite stable.

  • eeeeeee

    The big problem with firefox isn’t Firefox but all the crappy plugins and add-ons.

    Without FlashPlayer and all that stuff, Firefox is way more stable.

  • http://diary.braniecki.net gandalf

    Hi Jono

    So, as far as I definitely agree that the expectations are growing, and our userbase is growing, you base your post on the assumption that Firefox 3 is less stable than Firefox 2 was. Can you elaborate more on that? From what I know our data says the opposite. That Firefox 3 is the most stable and reliable version we released ever. Before Fx3 release we gathered over million people voluntarily testing betas which also has helped us a lot to make sure that it is prepared.

    Right now we’re preparing Fx3.1 which is a nice incremental improvement with features like or but I don’t know of any major concerns regarding stability, security, memleaks etc. that we had to fix because of some problems in Fx 3.0.

    So it’s hard for me to agree with your statement. Additionally, although it is true that we did employ more people, I don’t think that the number of employees working on Firefox has grown. Most of new hires are around Mozilla Messaging, Mozilla Mobile and revitalization of Mozilla Foundation. Additionally, majority of checkins are coming from volunteers which means that we maintain the community based model successfully.

    Maybe you are for some reason dealing with firefox bugs more often than average and you’re biased in the way that user support front line person has shifted perspective on how buggy a product is.

  • http://sites.google.com/site/thomasknowles/ Thomas Knowles

    With greater size, features and expected user base comes complexity. With complexity comes problems.

  • http://ubuntulinuxtipstricks.blogspot.co Mackenzie

    Or is it really that more money means more staff employed and instead the quality failure is due to management and coordination problems?

    You mean like the trouble with the Mythical Man Month? Too many places where different people’s code has to figure out how to interact.

  • http://chrislord.net/ Chris Lord

    So, that’s what explains the state of Ubuntu these days…

    ZING!

  • markrian

    Ooh, burn!

  • Roger

    The obvious point from where I’m sat is that the more money invested in a project, the larger the project is. You’re not going to invest £1 million in a 1000 line program… Greater complexity leads to more bugs.

  • Sean

    I don’t believe it has anything to do with money. It’s more about size and complexity. Firefox 3 is bigger and more feature packed than Firefox 2, which in turn has more crap than Firefox 1. Vista is many times bigger than XP, which is many times bigger than Windows 9x, which is bigger than most common Linux installs.

    Honestly, the worst application I’ve ever used in terms of stability is Evolution. And it is one hell of a massive, bloated application. It has feature I have never needed, but those are all loaded up all the time. It’s prone to crashes, lockups, and other weird bugs.

    Granted, I do realize Evolution had a lot of money poured into it. :) I also know that Apache has had a ton of money poured into it, but it is quite stable. It’s also fairly modular and simple, even if it’s not quite as petite as Cherokee or Lighttpd.

  • http://popey.com/ Alan Pope

    Ok Bacon, own up, how many millions did you sink into jokosher?

    :)

  • Bejamin Otte

    The correlations I would try to find is not the one between money and stability, but between a) how much the developers care about their product and stability and b) how much hired developers care about the product they work on.

    And then I’d ask what kind of people are hired to work on Mozilla, Ubuntu, Microsoft, Linux, X etc.

  • martin

    I don’t agree with Firefox being buggy at all (even though I have a ton of Flash crashes which sometimes seems to be related to pulseaudio, I dunno).

    About the money vs stability debate: I work as a full time closed-source (unfortunately, I’m looking to fix this asap) software developer and I also some some open source coding on my spare time. Basically it comes down to, at home I only do coding when I feel like it, when I care about what I’m building etc. At work this happen as well but at work I also have to do coding stuff that I don’t feel particularily interested in, old buggy problems etc and then I don’t care that much. Why should I spend hours of work cleaning up code that some else just p**ped all over? Most of the time I don’t get hours allocated for cleanup anyway so it would be evenings I guess. Not going to happen if I don’t care about the bits and I never use the work related products myself. At home I work on programs that I use every day, I work on programs that have really clean code (like GNOME and what not) and I deeply respect the skins of the GTK devs and so on so I do my outmost to make sure I only contribute top notch code.

    Commiting a single line of bad code into GNOME is like making a mess in your own kitchen, commiting bad code as work is like making a mess in a shared dormary kitchen (if the mess is bad enough, sure someone will make a fuss about it but you can get away with a lot of abusing of the commons).

  • JayBee

    :shock: I really don’t see any connection between money and stability.

  • Stoffe

    Does not match my experience at all. With Firefox that is, the short while I tried Vista is another story. FF3 is so much better than FF2, even though it is more feature packed. Your use cases may vary from mine, of course, and I think it’s likely it will boil down to us using different extensions/plugins and maybe also visiting different web sites with said plugins.

  • http://www.happyassassin.net Adam Williamson

    I think the problem is that Firefox started out as basically a two-man project to create a small, fast browser, and has somehow morphed into The Saviour Of All Humanity, and the organizational structure isn’t quite adjusted yet. :)

    A lot of us started out using Firefox when it was still Phoenix, and the Firefox that exists today is really not much like that project at all. I also think there’s a problem where there’s an inherent drive to make new versions of Firefox do Exciting New Stuff, which I don’t think is necessarily what everyone wants in a browser. I want it to do the Same Old Boring Stuff, more efficiently and with fewer bugs, but that’s not the project they’re coding.

  • bob

    So if Canonical stop paying you, will you stop talking so much shite?

  • http://ubuntulinuxtipstricks.blogspot.co Mackenzie

    Martin: Have you read The Cathedral and the Bazaar? You just described a chunk of it.

  • Rusty

    Mythical Man Month has already been mentioned, as has the Cathedral and the Bazaar. I would note that both have a significant counter example in the control software for the Space Shuttle.

    If you go through Feynman’s autobiographies, in at least one of them he relates his experience on the Challenger failure committee. Among the experiences he ran into was reviewing how NASA develops software for the shuttle. The computers are ancient by our standards, and the software being generated has to work flawlessly. It’s being developed in a closed source environment by a very large number of people. While the budget is hardly astronomical, you don’t get high quality programmers (which I dare say you pretty much have to have for the space program) by paying them at bargain basement pay scales.

    I think the key is that interface specifications have to be clearly defined.

    One of the problems I’ve seen with some applications has been that if you want to write a plug-in to do some sort of processing of the data, you have to go through the source code with each minor revision to see what the interface has morphed into this time. This becomes less and less of a problem as good quality software gets closer to a production release, but occasionally crops up between beta revisions as well. Essentially no one identified what that interface is going to look like for a third party developer, or user, in such a way that the project built to that interface. Instead someone decides that they need to massage the data in some way, then make that massaged data available to the third party developer if they want to do something else, and ‘oh by the way, we expect it to come back in exactly the same format.’

    I strongly suspect that with the exception of bugs in the code loaded by flash plugins, (or Java, python, svg, etc.) that is not the fault of the plug in developer, problems with failure to expose the developer interface and stick to writing to that interface, has been the largest problem with most project code. Open source or closed. And as projects get larger, that variety of problem is more likely to show up.

  • Andy

    Vista has not crashed for me yet, after about 6 month of use.

  • jono

    Some responses:

    Alberto Ruiz:

    Jono, do you have any sort of information to back that statement up? Which versions of Mozilla Firefox are you comparing?

    No specifics – I have just been picking up on an increased sense of instability when I use it. I don’t want to focus too much on Firefox though, like I say, it seems to happen to happen other projects and products too when more money is poured in.

    gandalf:

    So, as far as I definitely agree that the expectations are growing, and our userbase is growing, you base your post on the assumption that Firefox 3 is less stable than Firefox 2 was. Can you elaborate more on that? From what I know our data says the opposite.

    Interesting – what data have you been getting on stability, and how is that measured. To be clear – I am not doubting you :) – I am just curious to the metrics.

    Maybe you are for some reason dealing with firefox bugs more often than average and you’re biased in the way that user support front line person has shifted perspective on how buggy a product is.

    Very possibly; I am just speaking from my personal experience and don’t want to incinuate that my experience is the same as others. I have just generally experienced more slowdowns, crashes, and I have had to kill it manually a few times.

    Roger:

    The obvious point from where I’m sat is that the more money invested in a project, the larger the project is. You’re not going to invest £1 million in a 1000 line program… Greater complexity leads to more bugs.

    Indeed – this is my thinking too. It also reminds me of the law (which I can’t remember offhand) that debunks the idea of throwing more people at a problem to improve the efficiency of it being solved. I wonder if there is a typical current valuation cost that can be applied to a software project dependent on the level investment, and how that can be compared to a quality rating.

    Alan Pope:

    Ok Bacon, own up, how many millions did you sink into jokosher?

    1.2 :)

    Bejamin Otte;

    The correlations I would try to find is not the one between money and stability, but between a) how much the developers care about their product and stability and b) how much hired developers care about the product they work on.

    I agree, but surely this would plague much smaller projects as well. Or, do you feel that as projects grow in size, developers care less? I wonder if we could measure how much people care. :)

    bob:

    So if Canonical stop paying you, will you stop talking so much shite?

    Sorry no. I would probably actually produce more shite – Canonical ensures I have a job to keep me occupied. :)

    Rusty:

    I strongly suspect that with the exception of bugs in the code loaded by flash plugins, (or Java, python, svg, etc.) that is not the fault of the plug in developer, problems with failure to expose the developer interface and stick to writing to that interface, has been the largest problem with most project code. Open source or closed. And as projects get larger, that variety of problem is more likely to show up.

    Fascinating example with regards to the space shuttle, thank you for that. Your view here was shared by eeeeeee earlier in the comments. I agree – I think this is a big issue too. I should try running Firefox (as the example in the blog) without any plugins and see what happens. Mind you, it would not be a fair test as Flash is such a significant use case for a browser now, and particularly Facebook, which was a source of a bunch of FF bugs.

    So folks, so far we seem to have the following potential causes and notes:

    • Project size and complexity.
    • As the project grows, so does the amount of money invested, but the delta may remain the same.
    • Plug-ins and API stability can be a strong source of issues.

    Any others?

  • Zac

    Jono raises an important point, especially when you move away from the linux example. So, what is really being asked is: Does money=quality ? Does a CEO or an executive being paid $12million a year is worth a 150 times or more than an another employee? Much of the additional money that a company brings in goes straight to the pocket of those executives. There a many examples of executives increasing their income as the same time that they are laying off employees or trying to reduce their conditions. In this case a company is bringing in the money but the employees are reducing, this has a direct negative effect on quality. The quality of the companies products does not improve nor does the long term viability of the company. Past a certain limit money has a negative effect on peoples behaviour. To cut it short, money does not equal quality.

  • http://www.happyassassin.net Adam Williamson

    “So, what is really being asked is: Does money=quality ? Does a CEO or an executive being paid $12million a year is worth a 150 times or more than an another employee?”

    That’s not what he was talking about at all.

  • sulfide

    I have been wondering this same thing myself. I am using chrome in windows all the time now. I can’t keep up with the constant crashes or pages just plain stop wanting to load in FF3. I’ve always been a big fan too, it’s a bit sad. I still use it in linux though since there really are no good replacements for it.

    And to all the people who seem to think it’s plugins…I use absolutely no plugins etc in FF3, the only thing I am guilty of is tons of tabs open for days.

    I would love to know whats happening that the quality is sliding, and if money plays some factor. At the same time, when more money comes in and more engineers are hired…people may not be as familiar with the lay of the land and somewhat make things worse. Almost like the mythical man month says. Oh well, nice to see someone write about this who’s blog is under a constant watchful eye.

    I think some of the comments here people need to get their head out of the ass like that martin guy or eeeee

  • http://erwan.jp Erwan

    Jono: I don’t know what metrics in particular Gandalf was thinking about, but I know some of the metrics that Mozilla is gathering (in addition to user feedback of course): * crash reports * performance tracking; there are servers that fetch builds and run tests to catch performance regressions: http://blog.johnath.com/2008/11/26/performance-dashboard-v2/

  • http://decafbad.net Craig Maloney

    I think there’s some correlation, as the itches of the community don’t get scratched as much as the itches of the corporate masters. Evolution is a big sticking point for me in this regard, so much so that I wrote a blog entry about it:

    http://decafbad.net/2008/12/01/evolution-to-thunderbird-again

    Whether there’s a direct correlation between cash and stability decline may never be quantified, but the perception that certain companies are advancing their products to the detriment of the stability of those products is pretty self-evident.

  • http://www.pasicide.org Ciprian Mustiata

    Yes, I think there is. A company when invest money, they ask for visible features only. There is less important the QA step, even they invest in it. FF3 was build on better technologies. As Vista too. After crashes will affect as lose of data in some environments, the companies invest back in testing and QA.

  • http://linuxtent.com Ramy Eid

    I think more money means more risky changes to the software. Developers think more changes means more productivity and achievements. I am not sure if firefox is not stable, I think ubuntu is far aways from being stable. I spend 1-2 hours everyday to fix the wireless stuff for my friends. Because I am busy with the my own projects. I stopped to invite people to use ubuntu.

  • Justyn

    I know anecdotal evidence isn’t of any use to you, but anyway:

    I’ve been using Firefox under Intrepid for a month and pummeling it with 8-12 tabs open always. I have flashplugin-nonfree installed for the flash.

    I have not had it crash or lock up even once. I think this is the most stable Firefox I’ve ever used.

    Perhaps a naughty plugin is the cause of your problems?

  • http://linuxoutlaws.com Fab

    I think you might be confusing influx of money with product complexity. The more mature software becomes, the bigger and complex it will be and the easier it is to screw up. I don’t think this necessarily has to do with money. I think even the Linux kernel has been suffering from this, lately. Maybe it’s been kept in check better there because of more eyes looking at the code…

  • http://blog.zrmt.com andylockran

    On a budget, you’re forced to making decisions to cut features, and to minimise waste. Once you start getting more money, your software becomes (sometimes needlessly) more complex.

    It’s a trend that appears in so much Open Source Software. Rather than a big expensive unified blob of lost of stuff that doesn’t quite work, Open Source is built up of lots of lower-budget parts which do the one thing they do well, and stick to their functional requirements.

    Biased, me? Of course not.

  • http://www.lewiz.org Lewis

    I can’t believe nobody has brought you up on Vista being unstable. Since when? It’s more reliable than Ubuntu on my laptop

  • jimcooncat

    Too Big To Fail. When a project is young, the team has an entrepreneurial mindset. There are big risks involved in “investing” development time. As the workforce expands, management gets very mindful that the activity is becoming more precious to maintaining their lifestyle. If the product falters, they’ll be eating mac and cheese instead of steak.

    So Management takes the steps they can to limit losses. They add bureaucracy. This is seldom well-done in an efficient manner; management in and of itself is not subject to quality control in many firms. So the devs need a new box, they now have to submit their request and wait for management to wade through the new paperwork that was created to limit waste.

    As the paper shuffling increases, motivation drops. Management becomes nervous, and adds more requirements and regulation. The quality suffers increasingly because of fear.

    Contrast this with a project like Chrome, where the firm’s wealth does not hinge on the success of the browser. The bureaucracy itself can be controlled, or even torn down and rewritten from scratch without anyone going hungry. No fear — it’s liberating.

  • nnonix

    I think it has more to do with feature creep and excessive backward compatibility.

  • http://stuartaxon.com Stu

    A friend of mine has switched from FF3 to Opera, because of it crashing compared to FF2. I noticed it seemed to crash at least once a day, once I switched to 3. Unfortunately it wasn’t possible to trace what it was doing. I put it down to the big changes that happened in 3, so hopefully these will go away over time. I also notice my 4 core computer sometimes stalling when firefox is open, really can’t be sure what is going on though… something horrible and intermittent… possibly flash is killing things.

  • http://whereswalden.com/ Jeff Walden

    If you’re having problems with Flash, try running with Flashblock. I was crashing a decent amount in Flash code (I run official builds with talkback, now breakpad, reporting and can see crash stacks) and seeing high CPU usage when viewing sites running Flash, and Flashblock very quickly fixed both problems. Add exceptions or click through on the sites that need it, and it works fine. Only a few sites fail if Flash is click-through blocked (video.cnn.com comes to mind, boo!), so you don’t really miss anything — except rickrolls, if you prudently don’t whitelist youtube. :grin:

    This is just one demonstration that Firefox is part of a larger system; be careful how you assign problems. Reading this made me realize that I hadn’t crashed recently (since November 16) when I’d been crashing very often (11 times in three days!) from November 9-11, with stacks implicating gfx code and text rendering. Then I remembered that from bugwatching I’d heard of a number of bugs where Mozilla had to work around Apple’s font routines somehow corrupting system caches, resulting in general instability. That’s not your problem here, of course, but it’s worth keeping in mind: bugs seen at the top of a complex stack of technology aren’t necessarily the fault of the top level of it.

  • http://webbookblog.com Alan Bell

    Money leads to complexity, complexity leads to instability, instability leads to the dark side.

  • http://wolfger.wordpress.com/ Wolfger

    I don’t think it’s necessarily the money… It’s the people. Hmm. That doesn’t sound right. But what does the money do? Money doesn’t code. It hires coders. So when you throw as much money at a piece of code as Microsoft does, what do you wind up with? An army of coders, of varying quality, and they do not know what all the other coders are doing. They probably have a spec to code to, as all the other coders will as well, but I’ve never seen a spec that 2 people couldn’t argue over the interpretation of.

    So when money buys quantity of coders over quality of coders… and coders who feel (and no doubt are told) they must code for at least 40 hours per week (as opposed to volunteers who pretty much only code when they want to, and therefore enjoy what they do)… and the inevitable contradictory visions of coders causes problems… Yes, I suppose an excess of money (coupled with poor people management) can ruin a project.

  • Why ?

    You need more data. Any two points can be joined by a straight line.

  • jono

    sulfide:

    And to all the people who seem to think it’s plugins…I use absolutely no plugins etc in FF3, the only thing I am guilty of is tons of tabs open for days.

    Although this is the worst possible place for bug reports, I should share that I also have an absolute orgy of tabs open. Maybe that is the source of some of the problems.

    Erwan:

    Jono: I don’t know what metrics in particular Gandalf was thinking about, but I know some of the metrics that Mozilla is gathering (in addition to user feedback of course): * crash reports * performance tracking; there are servers that fetch builds and run tests to catch performance regressions: http://blog.johnath.com/2008/11/26/performance-dashboard-v2/

    Thanks for the pointer. Their measurements look interesting, but very performance focused. I would be interested to see how they make use of their bug archive and formalised QA testing. Again, I don’t wish to hammer on the Mozilla folks, we are all plagued with stability issues as projects grow bigger, I am just curious what solutions we can share. Thanks for the link, Erwan. :)

    Craig Maloney:

    Whether there’s a direct correlation between cash and stability decline may never be quantified, but the perception that certain companies are advancing their products to the detriment of the stability of those products is pretty self-evident.

    Hey Craig. :)

    I wonder whether this is the case. Arguably, using Evolution as the example, the problem there is that it is rather unmaintained now. I would suggest that the problem there is that not enough money is being invested in it.

    Fab:

    I think you might be confusing influx of money with product complexity. The more mature software becomes, the bigger and complex it will be and the easier it is to screw up.

    I am not sure. What about gcc? I would imagine that has got insanely complex in the last five years, but the quality has always been pretty good to me. Another example is GStreamer – as that has got more complex, it has actually improved in quality. From what I can tell, the level of commercial investment in GStreamer has remained largely the same. I expect Christian Schaller to beat me down now…

    Lewis:

    I can’t believe nobody has brought you up on Vista being unstable. Since when? It’s more reliable than Ubuntu on my laptop

    Wow. Is your laptop build from Alien technology? :P

  • http://jaysonrowe.wordpress.com Jayson Rowe

    Interesting points – here is a couple of more thoughts.

    One commenter mentioned how Vista is a huge chunk of code that can’t possibly be tested on every possible machine it will be installed on.

    This is true, but wouldn’t be so if MS was more open w/ their beta programs (or gosh forbid their code!).

    Now, onto the Firefox thing (there is a connection) – I think in a way the same holds true for Firefox because now-a-days a web browser is probably one of the most used applications on any system, and everybody uses their system & their browser totally different. In a way, I’m like Jono w/ the “orgy of tabs” (i like that way of putting it). Often, I’ll use tabs as place holders – I’ll tab something I want to read, but not bookmark, and often I’ll have multiple windows with these multiple tabs, and that’s when things really go funky. The way it ends up happening for me is I’ll for example, open up google reader, start tabbing some stuff, and then I’ll need to check my work email, so I’ll ctrl+n a new window for that, and new tabs will start to spawn, eventually perhaps I’ll go into launchpad to do some bug triage, and ctrl+n a new window for that (having forgotten about what I was reading earlier in google reader) – by the time I get back to google reader, I end up opening even more tabs, and yes, eventually FF will crash…

    The moral of this story, is don’t hate one someone who says FF can get unstable (like it can’t possibly do that), because it can – it’s all in how you use it.

    P.S. ATM firefox is using 785MB of RAM on my Ubuntu 8.10 64-bit system – still stable at the moment, but too much more stress, and I’m sure it’ll bomb.

  • http://erwan.jp Erwan

    Regarding the number of tabs you have open, there is a feature that appeared in Firefox 2.0 I think, the bfcache (back-forward cache). It’s caching the rendering so when you click back it’s very quick. One issue is that the last time I looked, the cache was only capped per-tab and there was no global max for the memory that cache can use. So, if you have a tab and you do navigate in those tabs (not just open/close), the memory usage can grow quickly. If you get to the point it has to page, then it gets sluggish.

    This is something you can tweak with hidden prefs, so you can trade speed for memory (or the contrary).

  • http://linuxoutlaws.com Fab

    @jono: Could it be something to do with the numbers of developers as well? Maybe it really is not the money, but the number of people that come with it. That would turn my kernel argument (more eyes, bugs more shallow) on its head. But maybe you are right and payed people just lack the passion that the pure pro bono hackers have.