Working on Technology at Startups

Richard Tibbetts has called me out for conflating Web 2.0 startups with all startups in my recent blog posting, “Google has a problem retaining great engineers? Bullcrap”. His complaint was that I was over generalizing from Web 2.0 startups to all startups.

He’s right, of course. The traditional “technology startup” by definition does have a large amount technology work that needs to be done, in addition to the business development work. However, things have changed a lot even for technology startups. Consider a company like Sequent Computer Systems, which started in 1983. At the time the founders had a key idea, which was to use multiple commodity intel CPU’s to create first SMP, and then later, NUMA minicomputers. But in order to do that, they had to design, build and manufacture a huge mount of hardware, as well as develop a whole new Unix-derived operating system, just to bring that core idea to market.

These days, the founder or founders will have a core idea, which they will hopefully patent, to prevent competitors from replicating their work, just as before. However, these days there is a huge selection of open source software so there is much less technology that needs to be re-developed / re-invented in order to bring that idea to market. From Linux and BSD at the operating system level, to databases like MySQL, Apache web servers, etc., there is an awful lot for the startup to chose from. This is all goodness, of course. But it means that most of the technology developed in a typical startup will tend to be focused on supporting the core idea that was developed by the founder. If a company is focused on databases, they probably won’t be interested in supporting me to do my file system work. Why should they? There’s lots of open source file systems they can use; one of them will probably meet their needs.

So while it’s obvious that you can do technology at large variety of companies, of different sizes, I don’t think it’s arrogance to say that there are certain types of technology that can only be done at Google, or maybe at a very small subset of other companies. I’m pretty confident, for example, that Google has the world’s largest number of computers in its various data centers around the world. That means there are certain things that don’t make sense at other companies, but which absolutely makes sense at our scale. In addition, Google’s business model means allows us to pursue open source projects such as Chrome and Android that wouldn’t make sense at other companies. With Android in particular, it means that multiple firmware updates for any given handset model, if it causes people to use the web more and drive more advertising revenue, makes sense and so it’s something we can pursue — in comparison to Nokia, which gets its revenue from hardware sales, so a firmware update that extends the life of a handset is actually a bad thing for them; better to force you to upgrade an handset every year or two.

So I think Richard misunderstood me if he thought I was trying to make the argument that Google is the only place where you can do interesting technical work. That’s obviously not true, of course. But I do think there are many examples of technical work which don’t make business sense to do at smaller companies, and startups in particular have to be very much focused on bringing the founder’s idea to market, and all else has to be subordinated to that goal.  And one of the most interesting developments is how the combination of commoditized and standardized hardware, and comoditized software in the form of open source, has changed the game for startups.

For most startups, though, open source software is something that they will use, but not necessarily develop except in fairly small ways.   Many of the economic arguments in favor of releasing code as open source, and dedicating a significant fraction of an engineer’s time to serve as a OSS project maintainer or kernel subsystem maintainer, are ones that make much more sense at a very large company like Google or IBM.   That’s not because startups are evil, or deficient in any way; just the economic realities that at a successful startup, everything has to be subordinated to the central goal of proving that they have a sustainable, scalable business model and that they have a good product/market fit.  Everything else, and that includes participating in an open source community, is very likely a distraction from that central goal.

12 thoughts on “Working on Technology at Startups

  1. Sure, although I consider Lustre to be an exception which proves the rule. (For those who are following along, Lustre is an open source clustre file system which uses ext3/ext4 as a back-end object store.) Lustre is one of the few technology startups which (a) is using a pure open source business model, and (b) which has not only invested significant resources into open source into a core file system, but also dedicated a large amount of engineering resources to get the features merged into the mainline kernel.

    That is really cool. Unfortunately, it’s also very unusual….

  2. i guess i’m wrong then, but i used to think this should happen to many normal (in contrast with dotcom-like) tech. startups …

  3. it is true open source software is something they will use, but not necessarily develop unless a way is quite small. because I’ve experienced such a case. 🙂

  4. Thanks for the thoughtful response Ted. I don’t think we have major disagreement, but at the edges I think you are still too negative about what engineers at startups can do, and what can even be in the economic interest of the startup.

    You are right that employing a maintainer for a relatively stable subsystem of the Linux kernel, or making linux filesystems a few percent better, is something which makes the most sense at large companies which can get leverage out of many applications of the improvement or expertise they get.

    In the case of Android, my understanding is that the original technology was developed at a startup. It may not count as a successful startup by some metrics, but much of the interesting engineering work was done there. Google’s contribution has been to scale the application of the technology, mostly through marketing and branding, again based on my understanding.

    Contributing to open source projects is something that certainly happens at startups, though usually not the big established projects. My company StreamBase has contributed back to around a dozen open source projects that support our business, from big projects like Eclipse to small ones like getopt, to niche ones like QuickFIX. We’ve even taken on core maintenance of at least one project we use heavily, Janino. Contributing to open source in a startup is often more about the individual personalities of the engineers than about a corporate strategy. But most startups are using open source today, and it often makes sense to contribute back rather than forking or being subject to the whims of the active maintainers.

    Some of the advantages that Google has, such as a very large number of computers, are becoming more generally available. Much like previous generations of technology companies with unassailable economies of scale, this one is unlikely to last. Others, such as a great brand and a reliable stream of revenue from various forms of advertising, are difficult to clone in a startup. But advertising revenue doesn’t automatically lead to good engineering or open source. It just happens to be the strategy currently being pursued by Google.

    The biggest reason I prefer engineering in a startup is focus. Because the company generally has only one product and only a handful of technical innovations, engineers can be more confident they are working on important projects which contribute to the core business, and aren’t in danger of being accidentally canceled, redirected, or ruined by a change in corporate strategy.

    Not every project or product at startups succeeds. But the success or failure, the longevity and the direction of the engineering effort are more visible and more in the control of the individual contributor.

    For you Ted, I think Google is a great fit. There are certainly many engineers who, by personality or area of interest, will find their best opportunity to do good work at a large company like Google. But for people who want the control, vitality, and opportunity associated with a startup, they should keep an open mind about the kind of work that can be done in that environment.

    In 2003 people told me database startups were too expensive to make economic sense anymore, and now it is one of the more active segments. The same may turn out to be true of open source development, or extremely large cluster management, or even filesystems, if people try.

  5. @6: The biggest reason I prefer engineering in a startup is focus. Because the company generally has only one product and only a handful of technical innovations, engineers can be more confident they are working on important projects which contribute to the core business, and aren’t in danger of being accidentally canceled, redirected, or ruined by a change in corporate strategy.


    Engineers should always be able to figure out whether their contributions and their project is one that it makes sense for their company to continue to fund. Maybe it’s easier at a startup, but it should be possible even at a huge company like IBM. It does mean you have to pay attention to the VP’s strategy presentations, and talk to sales folks about what customers are saying, and reading the company’s 10-Q and 8-K’s. I’ve always believed that all good engineers should be able to read a balance sheet and a profit/loss statement, and be able to have a fundamental understanding about business strategy. (There’s a reason why I took all of those Sloan school courses, and why I watch the videos from the Founders Institute, etc.)

    Maybe at a startup the probability is higher that you’re doing the right thing, since there’s less middle management to get in the way and miscommunicate what the VP’s/founders had in mind. But at the end of the day, if you don’t want to depend on others, it’s best if you take responsibility for your own career, and that means making sure you know whether you’re working on something important enough, and that you are providing large enough return on investment that both you and your project won’t get cancelled.

    Because if you don’t, even if you’re at a startup, you’re still at the mercy of angels and VC’s regarding whether or not your company gets additional round of funding, and if you’re a junior programmer, you’re not going to get a whole lot of control. Richard, you’re the CTO of Streambase, so of course you have a lot of control. But even there, your key focus must be the success of your startup; do you mean that you’ve never had to choose between forking an open source project (perhaps because it would take too much effort to rewrite the code to the maintainer’s strict standards) or take on some technical debt because if you didn’t ship a product before the next round of funding, the survival of your startup was threatened?

    One thing that I find really nice about Google is that all engineers are considered insiders; which means that every week at TGIF we get a lot of in-depth detail about what the company is doing. We also get summaries from the CEO of the quarterly board meetings and what conversations he’s having with the board. Yeah, it means that I’m subject to the trading window, but it’s nice that I don’t have to work as hard get the strategy information. (I had to work harder to get that information when I worked at a startup, actually.) Of course, if you’re a senior enough engineer that you’re a founder or on the operating/executive committee, it doesn’t matter whether you’re at a big company or a startup. 🙂

  6. Yes, that’s good advice for engineers, not followed often enough. I encourage all MIT undergrads to take some classes at Sloan (the business school), if only to learn that business is not magic and not that hard to understand. Many Sloan classes, like 15.390, are very open to science and engineering students.

    I had not previously heard TGIF described so positively. That’s a really great thing to hear about Google. I hope that level of openness continues to work and is spread to more large companies.

  7. @6: In the case of Android, my understanding is that the original technology was developed at a startup. It may not count as a successful startup by some metrics, but much of the interesting engineering work was done there. Google’s contribution has been to scale the application of the technology, mostly through marketing and branding, again based on my understanding.

    Well, Android (the startup) was purchased by Google in 2005, 22 months after it was founded. Two years later, the Open Handset Alliance was founded. The first public release of Android didn’t happen two more years later, in 2009. So two-thirds of the Android development (in terms of calendar time) before the first public release of Android took place after Android was acquired by Google, never mind all of the work after Cupcake release in April 2009. Furthermore, as you might guess, there was a much greater ability for Android to be staffed up once it was under Google’s stewardship, so if you were to measure it in terms of engineering person years, I think you’d find the vast majority of it was underwritten by Google.

    (A lot of people seem to forget how long Google has been working on Android; indeed, Google has been working on Android for a lot longer than Apple has been working on the iPhone. In fact, Andy Rubin has been quoted as saying that one reason why Google started supporting Android development was because there was a fear that otherwise, a single company would completely dominate the smartphone segment. A lot of people assume that he meant the Apple iPhone. But if you look at the timeline, it becomes very clear that doesn’t make any sense at all. Just think about that for a moment, and consider who was really big in smartphones in 2005….)

    At the recent All Things D conference, Andy Rubin, the founder of the Android startup (and Google VP these days) has stated that Android would not have survived under under its original business model as a startup. He also stated that Android is now profitable if you consider all of the ads driven to Google thanks to Android. Which brings up a really cool thing about the advertising-funded business model. It’s often not in Nokia’s interest to send an OTA firmware update, since they make their money by selling hardware. So they would much rather force the user by upgrading by buying a new phone, and throwing the old phone into a landfill, rather than investing software engineering effort in creating a new firmware update.

    Whereas if a company is making money via an ads-based model, they have a lot more incentive to improve the existing devices by shipping firmware updates that makes it faster/easier to browse the web. The more people browse the web, the more ad revenues the company will make, and that’s all good.

    To me that’s one of the best reason why a “Google Experience” phone is a much better buy than one of the phones that have tons and tons of crapware dropped on them. Companies like LG have to get dragged kicking and screaming into whether or not they are going to port Gingerbread to their existing smartphones. I think they’re going to learn the hard way that if they don’t, they’re going to lose sales to other companies in the future, to competitors who can make the customer feel that they are much better future-proofed with a device where they are much more likely to get firmware updates.

    It’s another example of how the ad-based business model has some very distinct advantages. And so hopefully companies will begin to see that custom mods aren’t necessarily “value add”, but really crapware that subtracts value, because it makes it harder for them to ship updates in the future. And because of the value of the ad-based revenue model, it’s allowed the Android team invest a lot more technology than I suspect would be the case with Andy Rubin’s original business model (which was a fee-for-service selling Android to handset manufacturers).

  8. It’s a good perspective on Android. Certainly other startups with a similar model of developing cell phone operating systems (Savaje for example) folded. Being bought early and scaled up by a large company is a fine model for startups. Whether the best engineering happens before or after the acquisition is an interesting question, probably depending on a lot of factors.

    Regarding the advertising supported business model, I find the article What If They Stop Clicking ( interesting. Remember that advertising is fundamentally about letting marketers pay you to help them tell people things they don’t know they want to hear. But if all the worlds information is organized (TM) then there may be considerably less call for advertising, because people will find the products and services they need without it. Or if we get targeting to be more precise, then only certain consumers may be eligible for ad-supported content, because no one wants to pay to advertise to others.

    I’m not sure if Google has a public or non-public perspective on that. But it is one of the things which makes me question a world where the best technology is advertising supported.

  9. The thing about the advertising market is that it is huge — much larger than most people appreciate. Just in the US alone — ignoring the Asian, South American, Western Europe, and other international markets — the total size of the advertising market is $160 billion dollars per year. Out of that $160 billion dollars, TV is maybe $53 billion, Newspapers is maybe $30 billion, Magazines is around $17 billion, Radio is $17 billion, and Phone books is $10 billion.

    Some of these numbers are amazingly out of whack. When’s the last time you used a phone book? And yet advertisers are spending $10 billion dollars a year on phone books?!? Ditto for Radio ads. If you take the percentage of advertising spending, and then percentage of time spent by consumers with the various different types of media, it just doesn’t add up. And that’s a huge opportunity for a company that is specializing in on-line advertising. (And maybe that’s a hint as to why Google is doing Google TV. Even if Hulu and all the rest block streaming shows on Google TV, the ability to search for content, have applications that control your DVR, etc., is amazingly powerful. And yes, there will be ads shown as you search for TV shows or movies on in the next two weeks that feature Kevin Bacon, for example. Google TV is set up so you can watch that content which Hulu, et. al blocked via your DVR or cable box.)

    And yes, people can wring their hands about how people might be often ignoring banner ads. OK, but what about TV advertising or ads in the Newspaper? How often are people paying attention to those ads? And yet advertisers are willing to pay quite a lot more money to TV and Newspapers than they are to online ads — despite the fact that with online ads, we can give them tools so they can see which ad campaigns are successful and which ones are less so — much more than they could ever do with traditional dead tree and TV advertising. And to the extent that we do targetted ads, the response rates are in fact much better. Google even has some pricing schemes where Google only gets paid not only if the user clicks on the advertisement, but if they actually _buy_ the product. And that’s something both the advertiser and Google are happy with, which means that people are doing a lot more than just clicking on the ad; they’re actually buying product, which means it’s proof positive that the ad worked!

    So in some sense, advertising is also a way of organizing information. The customer has a need, and a business has a way of meeting that need. If Google can use signals given off by the user — always with the user’s permission — to make guesses about what the user might be looking for, and enable the user to more quickly connect with the business who can meet that need, it’s doing both parties a service, and it’s a win-win-win solution, all the way around.

    Anyway, that’s my view of the world, and it’s not necessarily Google’s — I’m too low on the pay scale to speak for Google <grin>. But one of the reasons why I’m comfortable with Google’s long-term prospects is just looking at the total size of the advertising market, estimates of how much time people spend on-line, and the misallocation and inefficiencies in how companies spend their advertising budgets. Where there are misallocations and inefficiencies, there are ample opportunities for a company to make the world a better place and make money at the same time while fixing those inefficiencies.

    And given that the full year 2009 revenues for Google was $23.6 billion, and the total US advertising market is $160 billion, and that $23.6 billion is a world-wide number, since Google does business all over the world, there is ample room in the market for Google to grow. Not just by fighting over the share of online advertising, but also by growing the total size of the pie by converting some of the ad buys for traditional media to on-line media. (And of course, Google isn’t the only company doing this; companies like Yelp and Facebook and Groupon all have the same high-level goal, although their revenue numbers aren’t as big as Google, obviously.)

    This discussion also ignores the enterprise cloud computing play which Google is making, and which was a big part of the Chrome OS launch. If you watch the demo at All Things D, there was an awful lot that was pitched at CIO’s who don’t want to deal with the administrative and security nightmare of a traditional PC laptop. And that’s why Citrix demo’ed how a Chrome OS laptop could use a Citrix WebApp to invoke legacy applications like Excel and SAP R/3. But that’s a different market and a different revenue model than the advertising-based model used for Google’s consumer user.

  10. I have to say that this has been a very enjoyable and interesting read. Both the blog post and the conversation in the comments between Ted and Richard Tibbetts.


Leave a Reply

Your email address will not be published. Required fields are marked *