I was recently checking to see what, if any follow-up there had been from Sun’s ham-handed handling of the Open Solaris Trademark, and I ran across this very interesting comment from John Plocher’s Candidate Statement for the Open Solaris Governing Board:
“I also think there was a misunderstanding about what Sun desired when it launched the community (in part) to encourage developers to adopt and use Solaris. My take is that, while there *is* value in getting more kernel, driver and utility developers contributing to and porting the (open) Solaris operating system, there is significantly *more* value in having a whole undivided ecosystem based on a compatible set of distributions, where application developers, university students, custom distro builders and users are all able to take advantage of each other’s work.
Put these two things together, and you can see Sun’s predicament. Sun *wanted* a community that empowered application developers, but *got* a community aimed squarely at kernel hackers. Whether you see this as the “kernel.org -vs- Ubuntu” fight, or the “fully open -vs- MySQL model” argument, in my opinion, it all is simply a reflection of the above mismatched expectations.”
So that explains why it’s take three long years to try to get basic open source development tools (such as putting Open Solaris source code in a distributed SCM located outside of the Sun firewall) for Open Solaris. It was never was Sun’s intention to try to promote a kernel engineering community, or at least, it was certainly not a high priority for them to do so. This can be shown by the fact that as of this writing they still are using the incredibly clunky requester/sponsor system for getting patches into Solaris; setting up a git or mercurial server is not rocket science. This lack explains why Linus gets more contributions while brushing his teeth than Open Solaris gets in a week.
So if you run into a Sun salescritter or a Sun CEO claiming that OpenSolaris is just like Linux, it’s not. Fundamentally, Open Solaris has been released under a Open Source license, but it is not an Open Source development community. Maybe it will be someday, as some Sun executives have claimed, but it’s definitely not a priority by Sun; if it was, it would have been done before now. And why not? After all, they are getting all of the marketing benefit of claiming that Solaris is “just like Linux”, without having to deal with any of the messy costs of working with an outside community. As a tactical measure, astroturfing is certainly a valid marketing trick. But after three years, the excuse of “just you wait a little longer, we’re just trying to figure this open source community stuff out”, is starting to wear a little thin.
Furthermore, if (as John Ploncher claims) this was about “empowering application programmers”, why was it that Sun’s first act was to trumpet how wonderful it was to release the Solaris source code under a Open Source license? This only seems to make sense if the Open Solaris initiative was really a cynical marketing tactic to try to save Solaris from being viewed as irrelevant. If that was Sun’s intention, I think it is fair to say that from a marketing point of view, the tactic has been at least partially successful — although as John has admitted, the goal of creating a full community with application developers, university students, and so on, hasn’t materialized for Open Solaris. Sun has the dream; the Linux community is living it.
However, from business standpoint, I wonder if Sun will really be able to sustain their Solaris engineering team if they will really be doing all of the work themselves, and outside contributions continue at the rate of 0.6 patches per day. After all, the margins when you are selling low-cost AMD servers are much lower than when you are selling über-expensive SPARC servers. With Linux, we have a major advantage in that kernel improvements are coming from multiple companies in the ecosystem, instead of being paid for by a single company. And given that 70-80% of Sun’s AMD servers are running Linux, not Solaris, it’s not clear how Sun justifies their Solaris engineering costs to their shareholders. Furthermore, if Solaris on x86_64 were to actually take off, there’s nothing to stop competitors from selling Solaris support — except the competitors won’t have to pay the engineering costs to maintain and improve Solaris, so they would be able to provide the support much more cheaply than Sun could. So while Sun’s marketing tactics have kept Solaris alive in some verticals, I have to question how successful Sun will be in the long-term.
Update: I’ve posted a reply to Brian Aker’s comments as a follow up that would probably be interesting to those folks who are interested in the ideas found in this essay.
Update^2: John Plocher’s name is spelled with an ‘h’, which I had omitted. My deep apologies to him for getting his name wrong. I’ve fixed it in the post.
April 23rd, 2008 at 12:12 pm
First, thanks for your talks on FISL. I talked to more people about them, and people loved them.
Well, my crystal ball says that Sun had little option but to open source solaris. After you have a strong community-backed open source player in a market segment, like Linux, unless you have some other very strong product to tie-in, like Microsoft does with Windows, proprietary products tend to die, because the competition of a single company against a whole ecosystem is very unequal.
I think they are doing the best they can, and actually their only choice, but I don’t think it will be enough in the long run, because of network effects.
April 23rd, 2008 at 1:17 pm
Ah, so THAT’S what Jon’s been up to since I saw him a couple years ago. For what it’s worth I think Jon’s a great guy, and while I don’t ALWAYS agree with his ideas and politics he’s an extremely bright guy and generally motivated. Moreover, he DOES usually try to do the right thing.
I need to call him again next time I get out to Palo Alto nee Menlo Park (PGP Moved).
April 23rd, 2008 at 1:22 pm
Gaining kernel contributors and building an active community was never on Sun’s radar since beginning and outside people are only realizing it now.
Initially I was trying to understand why the community participation was so low and showed no signs of improvements and my posts on opensolaris.org mailing list (which sucks BTW, has no open source code available, does not allow you to change your profile and Sun people have got no care for it) received some enlightening elitist comments from Sun employees. Sun and their employees believe only they can engineer Solaris well and all other outsiders are idiot hackers whose contributions need to be carefully watched and filtered by Sun engineers. (That is why they have that unbelievably stupid contribution process.)
And on top of all that - all Sun engineers do is a “job” - fixing and working on things that matter to Sun’s customers but little else. No wonder things stall like crazy - few months ago I checked their bug database (which btw is useless - no technical comments, no details on how it was fixed and no other interesting information like you would find on kernel.org bug reports) and they had open bug reports from circa 1999 of the type “Linux is faster than Solaris on this benchmark” .
All I could derive from my little but hard-facts and end-results based analysis is that a) Sun was only interested in forcing a “all is working out nicely” while ignoring the fact that they did not have many contributions, improvements on x86 hardware support were dismally slow, no community member was able to do anything with the problems reported by many people trying to get OpenSolaris to run on x86{_64} hardware and in their opinion Linux was a bunch of hacks which they wanted to distance themselves against.
What is really sad is how much potential good is killed by stupid, baseless, ignorant, protective mindsets controlling the OpenSolaris thing. They never get how OSS projects work, why people contribute, why communities grow the way they do and how to work with communities. They think they will be able to motivate people for no reason other than to work for what fits Sun’s definition of right - for free.
Stark contrast to Linux kernel community - where differences in opinions are explicitly allowed, managed and nothing is denied based on single person’s beliefs or wants (case in point - KGDB), where people and organizations work on what they believe, where there is little conflict of interest and where contributing and knowing stuff is incredibly easy.
Heck - closer to Sun, the GPLv2 OpenJDK has made a lot more progress in the little time that it has had but ignoring it is well within Sun’s rights!
April 23rd, 2008 at 6:06 pm
Hi!
I believe you are hitting the nail on the “organic” vs “nonorganic” open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution… but I don’t believe either are really optimized at this point.
Optimization to me would lead to a system of “less priests” and more inclusion.
Cheers,
-Brian
April 23rd, 2008 at 6:54 pm
[...] Ted put together a great set of insights into what Sun may have been up to with OpenSolaris. Ted has a great way of cutting through the marketing BS and gets right to the heart of what’s going on. Obviously I completely agree with Ted on a few issues. Do any of these sound familiar? I think I’ve covered some of this before here, and here, and here. [...]
April 24th, 2008 at 5:20 am
I hate to say it, but it really does look like things haven’t changed much since Bryan Cantrill’s one line reply to David Miller’s message about just how Linux manages to outperform Solaris on some microbenchmarks on SPARC.
It’s not so much about the technical content, as about Sun’s attitude to non-employees working on running fast on their hardware; reading the message put me off Solaris for life.
If you’ve not seen the original mail, it’s at http://cryptnet.net/mirrors/texts/kissedagirl.html
April 24th, 2008 at 7:47 am
Derek,
I’m sure John is a great guy, and I wasn’t trying to show any disrepect towards him, just as I have a lot of respect for many of the other Solaris engineers as well. I might not be so enamored with some of Sun’s sales critters, and Sun executives, when they take the most expensive Red Hat “we’ll fix the problem ASAP 24×7″ and compare it with a Solaris “support by email” in a Total Cost of Ownership study, but that’s considered “all’s fair in love and war” when using analyst white papers for marketing warfare, apparently. Hey, if the pharmaceutical industry can ghostwrite papers for academically peer-reviewed medical journals that will be read by doctors deciding which drugs to give their patients, why not computer companies for I/T analyst companies? But I digress….
April 24th, 2008 at 8:09 am
My comment was long enough that I decided it might be better to promote it to a top-level post.
April 24th, 2008 at 8:24 am
[...] Friends « What Sun was trying to do with Open Solaris [...]
April 24th, 2008 at 3:17 pm
It’s still better than what IBM managed to do: no free software, no community.
Until IBM stops spreading FUD about sun open source efforts and actually starts doing something relevant for free software(db2? aix? lotus notes?) I’ll just say: show me the code.
April 24th, 2008 at 4:33 pm
Jophn,
IBM has actually donated a large amount of code to Open Source. There’s Eclipse, the open source Java database called Cloudscape, not to mention a huge amount of Linux kernel work done by members of the Linux Technology Center. The Linux Foundation recently did a study of which companies were contributing code to the Linux kernel, and IBM was #3 on that list. In previous years, IBM had been #2, and the worry was whether we were sending too many IBM’ers to conferences such as the Ottawa Linux Symposiums. So if you don’t know about the contributions IBM has made to the Free and Open Source world, you haven’t been looking very hard.
April 24th, 2008 at 5:40 pm
Maybe the real problem with Solaris is that it’s
created for one sole reason: to sell SPARC hardware.
Nothing else really matters.
April 24th, 2008 at 5:42 pm
[...] T’so did a good job of analyzing the current poor state of open source solaris as a community. He points to a number of community building and engineering failures (such as building a mercurial [...]
April 24th, 2008 at 5:52 pm
>> [On IBM FOSS] I’ll just say: show me the code.
Download Linux. Download Eclipse. Download JFC. Get a clue.
April 24th, 2008 at 6:34 pm
[...] What Sun was trying to do with Open Solaris » Thoughts by Ted A few things Sun did wrong in trying to build the OpenSolaris community (tags: community) Bookmark to: Hide Sites [...]
April 24th, 2008 at 7:01 pm
Most of the articles I see these days are from people who aren’t part of the community, as a result, they usually get some things wrong.
First of all, while the *ON* community (the kernel/core os community) may have a low number of contributors, that’s just one component out of many that makes up OpenSolaris. There is a common misconception among outsiders that OpenSolaris, like Linux, is just a kernel.
It’s actually far more than Linux; OpenSolaris is a complete operating system.
If you just looked at kernel community comparisons, you might be tempted to say that no one has succeeded. However, it would be the wrong thing to say.
It completely ignores the enormous strides in other Community Groups that have been made in the OpenSolaris Community.
* opensolaris.org has over a hundred thousand registered users.
* The Advocacy Community Group has sixty-seven registered user groups from Australia to Venezuela (and growing).
* New OpenSolaris-based distributions are being created (such as Milax — think Damn Small Linux).
* Many projects (other than the “kernel” project) have direct commit access to outside contributors and receive many contributions every week
I would also like to mention that I don’t work for Sun, but I have direct commit access to the code repository for IPS ( the new packaging system — http://www.opensolaris.org/os/project/pkg/ ).
While I’ll readily admit that I would like to see many things change, a lot has changed already.
So, as for the three long years? *shrug* I’ve been a contributor and follower of this community since the beginning. Happy Birthday OpenSolaris, here’s to another three!
April 24th, 2008 at 7:11 pm
Shawn,
You’re right that some parts of Sun seems to get it better than other parts, with the Open Solaris kernel team being a prime example of those that don’t get it. I talked to a few of the Sun open source folks who went down to Porto Alegre for FISL last week and listened to their presentations, and some of them did seem to understand what an open source development community was all about. So fair points that there are some portions of “Open Solaris” out in the userspace that are more open that the Solaris kernel.
By the way, didn’t you violate Sun’s trademark rules when you referred to Milax and other distributions as “Open Solaris-based distributions”? I thought Open Solaris could only be used to refer to the Project Indiana distribution, which hasn’t been released yet.
I think it is fair to point out that John Plocher, a Sun engineer stated in his candidate statement for the Open Solaris Governing Board, that in his view Sun didn’t achieve what they wanted in terms of gathering together a community of application developers, university students, etc. instead what they got was a kernel-oriented community. So at least one person inside Sun wasn’t willing to paint quite as happy a picture about the community outside of the kernel as you did.
April 24th, 2008 at 7:28 pm
“You’re right that some parts of Sun seems to get it better than other parts, with the Open Solaris kernel team being a prime example of those that don’t get it. I talked to a few of the Sun open source folks who went down to Porto Alegre for FISL last week and listened to their presentations, and some of them did seem to understand what an open source development community was all about. So fair points that there are some portions of “Open Solaris” out in the userspace that are more open that the Solaris kernel.”
It actually isn’t a question of getting it — talk to any of the kernel developers. They do get it. Sun, unlike other companies, is actually changing their internal processes to be external. This means that rather than put up an external source code control system and let people “have at it” — they’re actually trying to change things so that inside and outside users are using the same tools.
A big part of the delay has been that our community (not Sun) decided that we wanted those source code control tools to have certain characteristics:
http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/
The result of that process was to choose Mercurial and subversion as our tools of choice, with Mercurial being the one for the kernel.
However, that soon revealed other problems.
* First, Mercurial was deficient in a few areas. Community Members working with the Mercurial project have been working to address any issue, file bugs, and test. We now have a version that is suitable.
* Second, additional tools for Sun employees needed to be written to help them transition from Sun’s internal, proprietary scm to Mercurial. This also took time.
* Finally, documentation and tests had to be done with community members both inside and outside of Sun.
The short of it is that Sun’s Solaris developers do understand what needs to change. But change is difficult — ask anyone.
I’m happy to note that the external source code gate will be going live soon, now that the tools, etc. are all in place.
“By the way, didn’t you violate Sun’s trademark rules when you referred to Milax and other distributions as “Open Solaris-based distributions”?”
Cute remark, but no
There is a difference between authors of a copyrighted work using someone else’s trademark and someone else using it to describe a work.
“I thought Open Solaris could only be used to refer to the Project Indiana distribution, which hasn’t been released yet. :-)”
The Developer Preview has been released and has been out since last year. Two of them in fact…
“I think it is fair to point out that Jon Plocher, a Sun engineer stated in his candidate statement for the Open Solaris Governing Board, that in his view Sun didn’t achieve what they wanted in terms of gathering together a community of application developers, university students, etc. instead what they got was a kernel-oriented community. So at least one person inside Sun wasn’t willing to paint quite as happy a picture about the community outside of the kernel as you did.”
Well, if being a candidate gives you special status, I was a candidate last year, and I was nominated to be one this year but opted to code instead
So, in which case, let me use my “special status” to point out that, as a community, we’ve been trying to shift our focus towards users and away from kernel developers.
As for the happy picture? Again, I don’t work for Sun, and never have
I stood up at the OpenSolaris Developer Summit last year and stated how unhappy I was with everything moving slowly with the contribution process for the “kernel/core OS community.”
I will readily admit I would love to be able to contribute a lot more to the kernel community, but until the process changes, it’s slow to contribute for external folks.
Cheers.
April 24th, 2008 at 7:28 pm
Ted-
I’m confused about your statement:
—
This can be shown by the fact that as of this writing they still are using the incredibly clunky requester/sponsor system for getting patches into Solaris; setting up a git or mercurial server is not rocket science.
—
As an external contributor to OpenSolaris, I have access to the OpenSolaris mercurial repository, and I use that to submit patches. Maybe you missed that process?
b
April 24th, 2008 at 8:05 pm
[...] from /., Ted at thunk.org has compiled a rant post “essay” with regard to Sun’s OpenSolaris [...]
April 24th, 2008 at 8:22 pm
[...] All the fuss over Opensolaris is pretty fun reading. In 2006 I wrote this here entry in frustration. 2 and a half years later what has SUN done? FAIL. [...]
April 24th, 2008 at 8:33 pm
Brett,
My understanding from reading various lists was that large portions of Open Solaris were still inside the Sun firewall. Let me see…. ah, yes, here’s a quote from David Conroy, whom if I’m not mistaken is a Sun engineer:
Maybe things have changed since February? If so, please let me know, and I will be happy to stand corrected.
April 24th, 2008 at 11:28 pm
Hey there, Ted. My last name is actually Comay not Conroy - we met a couple of times several lifetimes ago at the IETF, likely during a PGP key exchange.
In answer to your question, yes, progress has been made. It’s certainly slower that I would like but you can follow much of the progress at http://mail.opensolaris.org/pipermail/scm-migration-dev/ where the tools (over and beyond Mercurial) are being finalized before we push the ON and SFW consolidations outside the firewall. Of course, there’s a great deal of OpenSolaris project work already taking place outside the firewall including all of the Project Indiana work.
As I’ve mentioned to Michael Dolan before, I’m happy the Linux community is content with their development model and there are many attributes of it to be admired. But rather than casting it in a pejorative manner of “organic” versus “non-organic”, I’d rather just say that they’re different now. Over time, we might end up being closer to one another or perhaps the communities will always be very different in structure. But I would rather celebrate what’s similar which is an enthusiasm about our respective technologies and yes, as engineers, a belief in open development.
April 24th, 2008 at 11:58 pm
Let’s not forget these are also the same reasons Roy Fielding officially resigned…
http://mail.opensolaris.org/pipermail/ogb-discuss/2008-February/004488.html
April 25th, 2008 at 8:27 am
Steven Destika writes wondering why it is so hard for Sun to accept even small things. The answer is simple: quality control. The internal process(es) that are used to ensure that when a bug is fixed, it doesn’t break and that the change is properly researched and tested aren’t being relaxed for external contributions. Just one of the many things that requires acknowledgement is where the change came from and what license is associated with that change.
To just put mercurial on the outside and let people have at it would be a big mistake. Although it might result in a lot of activity, it wouldn’t necessarily result in anything usable and could easily fall in a heap.
As an example of what happens internally, every change that is committed is tested for correctness with a rebuild of the source code for lint and basic problems. How do you recreate that kind of process for 100s or 1000s of people around the world? Or do you not do that?
The internal mechanisms that support Solaris’s development are far more complex than Linus’s “cat diffs | patch” or whatever. Questions need to be asked about if those methods are maintained or dropped and if they’re dropped, etc.
There are reasons why the bug database has very few fields that are visible to the outside world: often confidential information of various sorts is put in those fields. The effort to review all of the text, aside from just the description, would be immense. The common practice today is for people that work on a bug to include the evaluation or workaround (both invisible via opensolaris.org) in the description (visible on opensolaris.org) - but doing that is really at the discretion of the person working on the bug.
You are, however, right when it comes to one person’s opinions, etc. The entire voting infrastructure behind projects at opensolaris is a mess and should just be thrown out as it demonstrates a fundamental misunderstand of open source: people work on stuff because it interests them and want to contribute that so that others can use it, they generally don’t give a hoot about those that aren’t interested or feel it is of no use. When the voting goes away and opensolaris becomes more liberal, then the community will really have a chance of growing and becoming seriously worthwhile.
April 25th, 2008 at 8:53 am
Yes, I work for Sun.
Miserable to see IBM employees going to great lengths to pour scorn on a competitor’s sincere efforts to build an open source community. The latter is hard, the former isn’t.
Sun opened the source code, and while kernel hackers joined the community, the majority of the “Linux community” are people who never touch the source code. Hence, the need for a binary “OpenSolaris”. Such a rescoping of the project was, of course, controversial with sections of the existing community, as you would expect in any open source community.
John Plocher’s frank analysis of this problem was brave, greatly appreciated by all with the community’s interests at heart and, above all, open. That it is being used here as a stick to beat OpenSolaris with is pretty lame.
April 25th, 2008 at 9:37 am
[...] What Sun was trying to do with Open Solaris [...]
April 25th, 2008 at 9:46 am
Darren,
I don’t get it, but maybe that’s because I don’t understand the details of the Solaris development process. Just because you put the Mercurial repository outside doesn’t necessarily mean that you relax quality controls! The official Linux git tree is out on kernel.org without any firewall protection, but that doesn’t mean that anyone can push changes into the tree. Changes still have to be reviewed and dissected on the Linux Kernel Mailing List, and in general tested in a variety of testing trees before they are ultimately signed off by Linus or one of his delegates for inclusion into the official Linux tree. So you could just as easily move the Mercurial tree outside the firewall, but only allow a very small set of Sun engineers only have the ability to push changes into the Mercurial tree.
The only really valid reason I could see for keeping the Mercurial tree inside the firewall was if you still had some sources that hadn’t been released under the CDDL, and so you couldn’t make the Mercurial tree available due to copyright issues. But unless you can get the IP owners to agree to change the license, the better course of action would be remove those items from the Mercurial tree entirely. After all, even if you end up rewriting or replacing those device drivers, or math libraries, or whatever, the older, closed source version of the libraries would still be in the Mercurial tree.
And removing closed-source materials from a source tree shouldn’t take three years.
The bigger issue really is one of how quickly can patches be reviewed for quality control and then applied into the source tree. And here, it’s clear that Solaris engineers are busy, and that writing new features or fixing bugs, or whatever, is more important than reviewing external changes and getting them into the Solaris tree, as the patch rate of 0.6 changes/day very clearly shows. That’s a priority and resourcing decision that was presumably made by the Solaris Software Engineering Managers. It may or may not have been a correct one, but it’s somewhat at odds with various marketing claims made by a certain CEO about how Sun is absolutely committed to having an open developemnt process.
April 25th, 2008 at 10:01 am
David (Comay — sorry for mangling your last name),
I didn’t actually come up with the term “organic” and “non-organic”. Credit for introducing me to those terms goes to Brian Aker from MySQL, who mentioned them in a comment to this post, and as I mentioned in a follow-up essay, the term “organic open source” seems to have been coined by the Firefox development team.
I also didn’t mean for the terms to be perjorative, but I think it very much drives a very important distinction between a very company-centric development process, and one which has a very wide base of support from many different companies. And this is true within Linux as well; for example, JFS as a filesystem, regardless of its technical merits at the time when it was introduced, was largely a failure because nearly all of the development effort came from within IBM. In contrast, the ext3 and ext4 filesystem has a very wide base of company support, including IBM, Red Hat, Bull, NEC, and most recently, Sun (after the Clusterfs acquisition). Not only did this mean that ext3/ext4 was less subject to the vagaries of a single company’s budget process, but it also meant that more people within the ecosystem could support it and add extensions for their particular needs.
Of course, there are tradeoffs; when a single company controls what developers do, it can be easier to plan roadmaps, and ensure quality control, and so on. It may certainly be more “inefficient” to have a large body of developers, and we certainly do have our own sets of controversies when one body of code is rejected or passed over in favor of another. But ultimately, I strongly believe that having a vast, international, multi-company development community is far more important than the short-term advantages of trying to cling to control. Clearly Sun management feels differently, or at least has different priorities, than if I were running things.
April 25th, 2008 at 1:28 pm
Patrick,
I wasn’t trying to beat OpenSlaris-the-operating-system or OpenSolaris-the-community. If Sun’s intention really was to try to get a fan club of application vendors and university students, hey, that’s a legitimate thing to try to do. What I did target is that this isn’t exactly necessarily what some of the Sun Salescritters and Sun Marketing efforts (including Jonathon’s blog) have sometimes implied or outright stated. So if the goal is to have a MySQL closed development for the Solaris kernel, hey, Sun’s ball, Sun’s rules. But then maybe it’s not so great that some of Sun’s PR efforts try to paint the Solaris as being developed in an open community using open development processes.
As far as pouring scorn, at least I did so using facts which I researched as much as possible and pointing out gaps between Sun’s marketing message and reality. I am an engineer after all, so I have the engineer’s compulsion towards truth. In a few cases where I may have gotten things wrong, I’ve tried to correct it and admit that I was mistaken. In contrast, I have seen Sun Salescritters promulgate shortcomings about Linux that were solved 3-5 years ago, and use Total Cost of Ownership analysis which compared 24×7 phone support from Red Hat with Support-via-email costs from Sun, in an attempt to win back customers that have switch from Solaris to Linux. That I consider positively slimy (although others have told me that’s just par for the course and should be expected; the idealist in me in is still outraged) and I pledge that I will never to sink to that level.
As far as it being hard to switch over to Open Source development models, I’m going to have to call bullcrap on that. It really isn’t that hard. Especially for contributions to say, man pages to correct spelling errors, or to device drivers for which Solaris has no support today and for which Sun doesn’t sell machines for said device drivers (i.e., for various laptop peripherals, etc.). Quality issues shouldn’t be a problem if there is no support for device today. For changes to the core scheduler or VM, sure, you have to be careful; we’re careful in the Linux world as well! Changes might go into various test trees for review, comment, and testing for a very long time before they get integrated into the official upstream kernel. It certainly shouldn’t take three years to do this, unless there are people inside Sun who have been fighting the move every step of the way. But given what your CEO has said about Sun’s commitment towards Open Source and Open Source Development, I would think that shouldn’t have been a problem.
Certainly, if you had been more open with respect to device drivers submissions, I suspect one of the major key areas where Linux has a dominating advantage over Open Solaris, in terms of device driver support, would be much less of an issue.
April 25th, 2008 at 2:18 pm
Ted,
There’s a blog entry I wrote 8 months ago on the OpenSolaris community that you might be interested to read:
http://blogs.sun.com/avalon/entry/manufacturing_a_community
And also this (over a year ago now):
http://blogs.sun.com/avalon/entry/what_is_opensolaris_success
and oh, the spelling mistakes one finds when reading old blog posts… not to mention the spam’d comments. Ugh.
But anyway, I think they’re both quite strongly related to your comments and thoughts you’ve put into this.
BTW, I’ll add that OpenSolaris projects, inside Sun, seem to be like any other IT project that is in any company: often runs over time and over budget. Mercurial is being used in some places but not others, at present, in OpenSolaris. Maybe we shouldn’t always look for malice (and conspiracy) but rather just human factors in why it is taking a seemingly long time for some things to happen.
April 25th, 2008 at 8:56 pm
Darren,
I looked at your posts, and on the whole I agree with them. In particular, the focus on the bureaucracy of a community instead the actualities of creating a true community seems to be a misplacement of priorities. This is not the way true communities are formed:
If you want a bit of free advice, I wouldn’t concentrate so much on getting the college students. You need to take your community members where you can find them. This is where making people wait 2 months before a patch can be integrated is really in the send self-defeating, since many folks will lose interest and just decide to go elsewhere.
In fact, given the regular 3 month release cycle for Linux, sometimes people will need to wait 2-3 months before a patch gets integrated, in the worst case. The difference is that it can get merged into a testing tree before that, and people who want can in fact run kernels based off of patches in the test tree. Take a look at what Andrew Morton does for Linux development; it’s amazing to me that the Open Solaris kernel folks haven’t taken some of the good ideas that we’ve developed over 17 years of open source development experience. Not all of them may be appropriate for Solaris, but there are some really basic things that you could do that would make a huge difference.
April 25th, 2008 at 10:05 pm
On OpenSolaris…
In a recent article, Ted T’so makes some interesting points on Sun’s motives behind OpenSolaris, and how it fares today in the FOSS ecosystem as a result.
“Fundamentally, Open Solaris has been released under a Open Source license, but…
April 26th, 2008 at 2:53 am
hi, Ted …
Our communications on the OpenSolaris project been spotty at best. From a company perspective, from a marketing perspective, and from an engineering perspective.
As a result, some people think we built a community for kernel developers only. Some think the community is primarily sys admins. Some think the future will mostly be for users. And still others think we set out to build something only to end up with something else (John’s not the only one who feels that way but it’s a view I disagree with). From my perspective, and I’ve been on the project team from the beginning, we had /always/ planned to build a multi-level community that included a wide variety of people — sys admins, hackers, individual developers, corporate developers, students, writers, us ers, kernel/ app/driver developers, distribution and appliance builders, customers/partners, etc. I think gamers were on that original list, too. The list was way too big, obviously, but that was pretty much what we talked about before we opened. But although the /intention/ to build a diverse community has always been there, we’ve had to deal with many constraints and dependencies along the way and we couldn’t engage all these levels of the community simultaneously. That’s why different people see the community differently. It /is/ different depending on when you view it and how you engage with it. Vastly differently, actually. Also, we are not talking about opening a small side product from the corner of a very big company. Instead, we are talking about opening the company’s /core/ product that touches virtually every part of the company and our customer base. It’s clear the business and technical complexities have been more complex than we originally thought. Some people have called our plans “half baked” and other such pejorative terms. That’s unfortunate. I’m not sure there is a previous example of a company this size opening a source based this big that pervades a core business like Solaris does for Sun.
However, our initial plans were clear with this one point — the opening of Solaris (code, infrastructure, process, people) and the building of a global community would have to be a multi-phase, multi-year program. There was no other way given the circumstances, and that should not surprise anyone who has any awareness of the scale of this project or the challenges of the project team at Sun. Heck, it took two years just to release most of the code, which was challenging at times due to some of the historical issues involved in this source base. But during that time we’ve been out there building a diverse community. It’s small by Linux standards, but we are young and many of us look to Linux as a great example of a thriving community. Keep in mind, we started with zero. But we have been slowing building and people have been coming and contributing in a variety of ways — although we /all/ realize that contributing to OpenSolaris has not been easy. Also, new projects like Indiana (which was not even possible until very recently) will increase the number and scope of contributions, which will increase the size of the community. So, in my mind, the OpenSolaris community already has much of the diversity we had hoped would develop. The seeds are there. I’d say it’s probably accurate to describe us as technical community right now, but I don’t think we are only kernel developers, and I don’t think we set out to build something only to find ourselves with something else. I think we set out to build something, and we are still building.
None of this excuses our mistakes in implementing or communicating. We’ve made those mistakes and hopefully we’ll learn and get better.
April 26th, 2008 at 2:35 pm
Still haven’t got the name spelled right …
> Furthermore, if (as John Ploncher claims) t
April 26th, 2008 at 3:43 pm
Thank you *so* much for clearing this up! I’ve been reviewing Open Solaris releases for a couple years now, and been baffled by the apparent lack of progress. I am, indeed, guilty of mistaking intentions and having false expectations. When I heard the news that Sun was opening Solaris and porting it to the x86, my interpretation was “Oh boy! A FOSS Unix just like Linux, but with Sun backing it!” OK, I’ll stop expecting that now.
Finally, this post answers some of the questions I’ve been asking.
April 26th, 2008 at 6:18 pm
I think this is an excellent post and an outstanding discussion on a topic of great personal interest.
I see a certain degree of agreement from both “sides” in this discussion:
1.) OpenSolaris is a flawed effort that did not come “out the door” in the best possible way and is still far from perfect, and
2.) It’s improving, although it is taking a lot of time.
I was a Sun employee, one of those dreaded “marketing people” I might add, and you can still see signs of my craft littered about the internet. Because of my past associations with Solaris, I have a slightly different view of all this.
OpenSolaris is not growing to the bazillions of participants kernel.org has for the simple reason that there’s no compelling reason that there is very little compelling reason for the vast majority of people to care about Solaris. Solaris may be a more stable, easily-managed desktop environment, but Windows XP, Linux, and MacOS X are “good enough” (I would argue that MacOS X is better than Solaris on the desktop, and that Linux is at least just as good). Solaris may be a better small server platform, but Linux is certainly good enough (maybe better for simple things). Finally, you’re up to large servers, and Solaris is clearly a much better platform for them, but there’s a very small number of people who care about large servers. Furthermore, they’re just about all in very large companies that care much more about business management than kernel hacking. Each of those companies may have a couple of people who pay attention to OpenSolaris, but it’s really not important to them. There are some visionary folks who see ways to leverage (or potentially leverage) the OpenSolaris platform in interesting and unique ways, but not very many of them, either. So the OpenSolaris platform remains relatively small and pretty Sun-centric.
For those who think that Sun’s management has dark intentions and sinister intent in OpenSolaris, let me assure you that the majority of Sun’s management couldn’t find a dark intention or a sinister intent with both hands and a flashlight. The vast majority of them spend all their days poring over spreadsheets and Gantt charts, and trying desperately to understand what Jonathan’s latest blog post means to their budget allocation for next quarter.
Now, there were two things that my (admittedly demented) sense of humor caught that might well go unheeded, but I won’t let them:
1.) Did an IBM guy say that SPARC is “expensive?”
2.) Bryan Cantrill is kind of a petulant child and an arrogant butthead, but, even when I have been the target of his rapier wit, I have had to admit he is one of the funniest people around. C’mon, folks, “Have you ever kissed a girl” is pretty damned funny!
My opinion, freely offered, and worth what you pay for it.
April 26th, 2008 at 7:22 pm
Bill,
Did an IBM guy say that SPARC is “expensive?”
IBM sells much more than Power machines, you know. It also sells Intel/AMD class servers, under the System x brand. Linux is interesting in that it is the one Operating System that works across all of IBM’s hardware architectures, from System x, System p (Power), and System z (Mainframe — the ‘z’ stands for zero downtime).
You’re almost certainly right, although, I haven’t done the benchmarks myself (the Solaris license doesn’t allow it :-), that Solaris probably scales better to the very biggest, super expensive Sparc mainframes than Linux does. The same is true for AIX on the Power architecture. And there’s a very simple reason for that; the margins are the highest on the really big machines, and so there is a huge incentive to make sure that the primary OS for said large servers can take best advantage of the million dollar and up high-end servers. I’ve been told that one of the reasons why Linux beats the pants off of Solaris on small servers is because Sun engineers were told to spend time optimizing for the high end machines, and that makes economic sense, even if 99.99% of their customers were using much smaller-scale machines. And that may certainly explain why some customers have made a point of calling Sun “Slowlaris” (and perhaps why the Solaris EULA prohibits benchmarking :-).
Solaris may be a more stable, easily-managed desktop environment
I’ve got to ask — why do you think that Solaris has a better desktop environment?
April 26th, 2008 at 8:03 pm
Ted-
I just enjoy yanking on the “expensive” chain. The simple truth is that all the servers in each class fall into a pretty narrow band of price-performance, and each type is better in some wayv(e.g. Power machines are WAY better at floating-point than SPARC or x86, etc.) The marketing folks (not the talking heads like me, but the people who spend a lot of time sweating over hot spreadsheets) make very damned sure of that. IBM, Sun, and HP are all very cost-competitive; to call one or the other “expensive” is just silly. If you’re comparing a 2-plug x86 to a 2-plug SPARC or Power6 machine, the SPARC and Power boxes look really expensive, but there are attributes that make them worth that price premium to some customers.
Please believe me when I tell you that Benchmarketing (i.e. running benchmarks for the sole purpose of telling the world that one architecture, machine, or piece of software is better than the other) is a task best left to professionals. It’s one thing to point out that one kernel performs a given operation 20% (or 3 times, or whatever) faster than another, and quite another to extrapolate that to something that could be meaningful to a customer making a dollars-and-cents decision. When you start using blanket statements like “Linux beats the pants off of Solaris” you do so at the peril of your credibility.
Again, please trust me on that…
People call Solaris “Slowlaris” primarily because it’s funny. Which, of course, I embrace - funny beats accurate every time!
And (I’m kinda surprised you missed this, since you quoted it) I didn’t say that Solaris has a better desktop environment (I actually said that MacOS X and Linux had a better desktop environment) - as you quote, I said Solaris may be a more stable, easily-managed desktop environment. Of course, those aren’t the most important quality attributes in a desktop machine, usually.
Thanks again,
Bill
April 26th, 2008 at 9:42 pm
Bill,
I’m well aware that Benchmarketing is definitely left to the experts — because they can come up with results that have nothing to do with reality, never mind customers trying to make a dollars-and-cents decision. I’ve talked to a number of customers who were quite serious about Solaris being slower than Linux on their workloads. I’ve even participated in head-to-head benchmarking wars where I helped tune Linux systems to beat out Solaris on-site at a customer (this is where both Sun and IBM had teams on-site and usually a fixed time to deliver a better performance and/or better price performance on a customer workload). And yes, there are workloads where Linux did lose out, but they generally tended to be on the higher end machines — and more often than not, it was because Solaris had done finer grained locking than Linux had in some particular system call or VM code path. (And as we all know, finer grained locking penalizes you on uniprocessor and small SMP boxes, but is critical for performance on large SMP boxes; see again my observation about how the traditional Unix systems tend to optimize for the manufacturer’s higher margin boxes, even if it penalizes the low-end.)
You didn’t say Solaris has a better desktop environment? OK, so why do you think Solaris is “a more stable, easily managed desktop environment”? Especially in terms of “easily managed”, once you get to Project Indiana, both Linux and Solaris are using mostly the same upper-layer desktop software stack, so I’m not sure where you get “more easily managed” from. I’m just curious to hear where you think Linux could be improved.
April 26th, 2008 at 10:30 pm
I got somewhat interested in Solaris again after several years of having been exposed to it to try and work out what on Earth was going on with OpenSolaris. I’ve had the usual and predictable spiel recently from various Sun marketing people as well where it’s impossible for you to tell them that no, Linux is not unstable, and no, I’m not throwing out RAID, LVM, ext3 and XFS because ZFS will solve all my storage problems.
What I’ve found is that absolutely nothing has changed since Bryan Cantrill’s mail posting from twelve years ago. Whilst amusing, it was painfully obvious that Bryan didn’t have the faintest idea how to reply, and that’s the way things are. Within Sun there is still a strange mindset that, regardless of anything that gets discussed, Linux is merely an upgrade path to a mature OS in Solaris, and even now, x86 is merely a stopover on your way to proper SPARC hardware. I’ve experienced this kind of attitude with Sun before over things like Cobalt and their previous forays with x86 servers, and it makes me and a lot of other people distrustful of them still. It gets even weirder when you realise that Sun is selling a reasonable amount of x86 based machines, which are sales they would have once lost, and about three quarters of those servers they sell are running Linux, even though that has saved them from ever decreasing revenue and profit.
As for OpenSolaris, I just cannot for the life of me understand what’s going on there. It’s said that Sun wanted a community of application developers and what they got was a kernel hacker community. From where I sit, I can’t see that they have got either of those two things, and won’t get them in a month of Sundays. If you’re getting 0.6 patches per day, and it takes forever to commit a patch to anything, then you don’t have a kernel development community. As for an application developer community, it was obvious Sun needed one years ago. Linux users are spoilt with apt, yum, yast and various other package management tools that allow them to upgrade and patch systems quickly and easily, and install software quickly and easily. Doing that on Solaris, still, is an exceedingly painful process, and much of the software you will build on Solaris is open source software designed for a GNU toolchain. Even now, getting a GNU toolchain on a Solaris system is like smashing stones for blood (/usr/sfw/bin looks the same now as it did years ago), and you’re doing all that to install software that can be put on to your average Linux distribution in a few minutes maximum. The logical answer is probably not to try.
I can’t understand the OpenSolaris trademark thing. Sun have a site called OpenSolaris, but you can’t call the derivative OpenSolaris? What the hell is that all about? It’s like rearranging deckchairs on the Titanic while the band plays some ragtime tunes. I’m not going to try and understand it in all honesty.
The thing is, if Sun is serious about Solaris surviving then it needs an application and a kernel developer community - especially an application developer community. They desperately need a sane packaging system of some sort so that a wide variety of open source software can be installed, tested and generally supported. Particularly in academia, this is what has made a lot of people just plain dump a lot of expensive SPARC and Solaris systems for x86 and Linux. It’s cheaper to switch than troubleshoot. If you install MySQL, Postgres, Python or something else on a Linux based system, the simple fact is that it is far, far better supported there in terms of people testing and documentation than Solaris. Ahhh, documentation. Have you tried to find anything on Sun’s site?
Case in point. In a university a few years ago some pretty serious threading issues were encountered when running Python and Zope on a Solaris and multi-processor SPARC platform where requests would be served one at a time. Now, in a community setting where source code is open throughout the stack, that kind of problem could be troubleshooted in a collaborative manner rather than in the dark. As it was, Sun isn’t interested in Solaris running Zope, Python or indeed a great deal of open source or other software well, the wider community simply weren’t interested in troubleshooting a closed source system (probably because the people writing the software don’t run it), and it was cheaper to move to x86 and Linux.
If you’re laughing at the name Slowlaris, you don’t know the half of it.
In terms of the kernel developer community, it seems as though an edict has come down from JS about making Solaris open source, but the people implementing it are not motivated to make it work nor do they see any benefit. In reality, Sun needs a kernel hacker community if they want Solaris to be relevant or survive. Maintaining an operating system, a kernel and a bunch of drivers is an expensive thing to do. Devices exist from high-end servers to low-end NAS systems, and the fact is that only Linux can bridge the gap between them today. It’s a huge ask for a proprietary system, or a proprietary system trying to become an open one, to do that. Even AIX will probably get consumed by Linux at some point, as Linux becomes cheaper to maintain and port to newer architectures. Mind you, if Sun truly did what they say they would with OpenSolaris then it’s entirely possible that many companies could provide cheaper support for Solaris than Sun!
As it is, Solaris still takes hour after hour to install, it takes an absolute age to boot up, you don’t get a decent set of modern userland tools, you don’t have a toolchain that will compile and install most open source software, you don’t get a sane package management system, installing software over multiple systems is a nightmare, you don’t get virtual terminals……. The list is endless. All OpenSolaris can possibly hope to do is get Solaris into a reasonable state for people to download and use, solving the problems that Linux distributions (and BSDs) solved years and years ago - using much of the same software and replicating a lot of functionality.
Given that Sun are obviously calling the shots over OpenSolaris’ direction and its governance, I cannot see why any open source developer in his/her right mind would want to contribute anything (not just code) to OpenSolaris or what they would gain. There is an implied reciprocation that something will be given back when an individual contributes something to a project, and that just isn’t there with OpenSolaris. Goodness. Even the name is a problem before you’ve even got to any source code.
April 26th, 2008 at 11:02 pm
Segedunum,
To be fair, Project Indiana is supposed to solve the packaging problem, and the reviews that I’ve read seems like it is trying to create an “Ubuntu-like” desktop and packaging experience for Solaris. I’m not sure why it would be superior to a Linux desktop, and Solaris still suffers from the problem that (as you’ve pointed out) most open source programs are tested in Linux and less likely to be tested on Solaris, and thus it might not even compile on Solaris without modification. (I remember when the default was that most sources downloaded from the Internet or Usenet worked on SunOS by default, and it was a crapshoot whether it worked on HPUX or Ultrix; my, how times have changed.
That’s fundamentally a network effect, and simply being “just as good” as Linux isn’t necessarily going to help Solaris win over new users.
So Sun is trying. Whether they are trying hard enough, or enough to matter, only time will tell.
April 27th, 2008 at 2:11 pm
“So Sun is trying. Whether they are trying hard enough, or enough to matter, only time will tell.”
I actually disagree Ted - I don’t think it’s “trying” - it’s a conscious investment decision to under-invest but over-market and oversell a “nice” story publicly without really leveraging a distributed development investment model or giving up any control. I think you were subtly implying a similar point, but thought I’d an my view…
April 27th, 2008 at 9:48 pm
Mike,
You may very well be right. However, while I can certainly see Sun very clearly under-investing and over-marketing its Open Source development efforts, I don’t have proof one way or another whether it was born of malice or incompetence. So I was giving the benefit of the doubt by applying Hanlon’s Razor: “Never attribute to malice that which can be adequately explained by stupidity. ”
What Sun’s motives are doesn’t really matter all that much, though. What’s more important that people understand that net result is that Open Solaris doesn’t have most of the advantages that a project which has a broad and diverse open source development community. If it makes people feel good because it just happens to be released under an open source license, well people are entitled to their illusions. Just having an open source license doesn’t mean much, though, if there isn’t a development community to go with it — and that community is getting strangled by Sun’s control freak issues.
April 30th, 2008 at 6:56 am
I contributed a very minor library routine - something that other OSs have,
and has annoyed me that (Open)Solaris didn’t. Some discussion had
reminded me of it, I mentioned that there was already a consensus (and
draft standard) function that did the job, and someone said so why not
submit it? I did, as much for the exercise as anything else. Some hoop
jumping to be sure, but nothing beyond what I’d chalk up to sound
engineering discipline. Good sponsors help. I don’t recall it taking three
months start-to-integration. And it didn’t take long for other code to start
using that function as appropriate.
As a user, I strongly disagree with your earlier statement “Quality issues
shouldn’t be a problem if there is no support for device today.” Even
on a mere home desktop (nothing compared to a multimillion dollar
financial institution, or a nuclear power plant), I may be doing enough
different things at once in terms of home infrastructure and my own
services, that I’d want at least a clear distinction between what was
near or at production quality, and what wasn’t (or wasn’t known to be).
There has been recent discussion about different levels of hoop-jumping
corresponding to different levels of integration vs aggregation, or support,
or whatever, possibly even to be reflected in the eventual repositories,
so that one could distguish between supported/unsupported or stable/unstable. It’ll doubtless take awhile for everyone to agree on
the taxonomy and procedural changes, but the mere fact that it’s being
discussed means that people know that a single track process can’t
be responsive to a range of different expectations. And I think that’s
bound to extend from integration or aggregation process on one end, to
which packages in a repository are or aren’t eligible for support on the other
end.
Another point is that it’s not all Sun folks supporting bureaucracy and
all external folks wanting everything to happen faster and easier. There
are a number of external folks, who, while they might want some minor
changes, are at least as conservatively inclined as any of the Sun folks,
having used Solaris since the days when it did have a quality problem.
I fall in that category, and I know I’m not alone.
Finally, I was looking at some pages that give history on Sun’s quality
obsession with Solaris, and I just realized something: this isn’t about
building a new community, nor about corporate vs organic, it’s about an almost-open-borders policy being applied to an existing sizable but very
tight-knit community with an established set of values. Inevitably the
immigrants will be at a disadvantage until they understand the language,
and that the existing values have worked well enough to build something
they want to come to, and have no less reason for being there than those
that they bring with them.
May 3rd, 2008 at 9:58 pm
[...] some of the new features will not be available for the open source version. Open source people are suspicious about Sun’s intent. Personally I think Sun will not be able to get what they want, unless [...]
May 6th, 2008 at 8:38 am
[...] What Sun was trying to do with Open Solaris | Thoughts by Ted 一篇值得读的介绍open solaris的帖子.不管怎么说,linux始终在我心中占据第一位,其次是freebsd,然后才考虑其他 (tags: open-solaris sun linux) [...]
May 7th, 2008 at 11:42 pm
I would like to know what Sun is trying to sell: an OS (Microsoft)? A platform (as in the Mac)? A solution (like the IBM ads imply)? or Support (RedHat anyone)? That may help me understand what is the deal with Open Solaris.
May 8th, 2008 at 4:14 pm
Follow the money. They charge for full access to the sunsolve (patches,
bug reports, suggested workarounds, etc) site, but not for a copy of the OS
(either OpenSolaris or the commercial Solaris). And of course they sell both
hardware, hardware support, and esp. for larger systems, pretty much
end-to-end “solutions” too.
So among your alternatives, I’d say that right now, they’re _not_ selling the
OS (at least for smaller systems), but to some degree or another, they’re doing
all the rest.
They did sell the OS at one time; but that was probably more than 10 years
ago.
IMO where their history is mixed is in effectively _communicating_ what they’re
doing. I _think_ they’ve gotten the message that even if big iron is where the
big markups are, that desktops and universities are the mindshare that
provides tomorrow’s customers; there’s certainly been a lot of improvements
on that front. That doesn’t preclude them from doing silly things in the future,
but since there were alternative distros not long after they opened up
enough of the code. Now, there’s even at least one that at least aspires to
being commercial (the unlimited version of Nexenta). So at least on x86,
development needn’t strictly depend on Sun, although I think they’d have
to stray pretty far before it would be practical for someone else to try to fork.
July 28th, 2008 at 11:19 am
[...] coined by Brian - who was the highlight of the panel, in my opinion - the organic and inorganic terms describe [...]