Re: OpenBSD Benchmarked... results: poor!

[Bert de Jong <bert@ohreally.com>]

I read the tests and many of the comments on them.

I was very curious about the test, but very disappointed with everything after 
I read it.

The person who performed the tests was very naive when he said this ought to 
be the end of flamewars about network performance/scalability among the 
Linux/*BSD users. His test was just not good enough to be meaningful to 
anyone in the respect of offering acceptable conclusions.

A proper test would be

- Performed on proper hardware. I mean, doing a network performance test on an 
old laptop with inadequate harddisk size, with four very much server-oriented 
operating systems, one of which (OpenBSD) couldn't even be installed on the 
same part of the harddisk as the others! That's like organising a race 
between a Lamborghini, a Lotus, a Porsche and a Ferrari in downtown New York 
during rush hour, and disqualifying the Ferrari on the grounds that it is 
difficult to park backwards.

- Unbiased. This one was very much biased against OpenBSD, and in favour of 
Linux/'Leanux', as follows from many of the comments made by the tester.

- Performed by somebody who knew enough about installing and running all of 
the OS's involved to run tests on them (the tester seems knowledgeable about 
Linux, but is totally clueless as far as for instance OpenBSD is concerned.)

- Performed with a test programme that wasn't developed with a bias toward one 
of the OS's in the test (the test programme involved was developed on Linux, 
later 'ported to' *BSD.)

- Described plain fact, by someone who would be objective and who would avoid 
showing emotions about the subject, much less a general favour or disgust 
toward the OS's tested. This is especially important because of the sensitive 
nature of the test subject. There are so many flamewars already!

- Described withoud prejudice even /if/ the tester happened to be more 
familiar or friendly with one of the OS's tested. That would really help 
improve the value of the test. Unprejudiced==scientifical==professional==a 
virtue.

- Carefully giving minute details about the test conditions (hardware, 
software, test programme details, OS installation details..) This test wasn't 
remotely accurate, look how it even fails to mention at what time 
OpenBSD-CURRENT was downloaded. That's crucial information.

- Compare equally. The test is already invalid because it compares an ancient 
-STABLE NetBSD with a -CURRENT FreeBSD.

- Be clear about its subject. The test focuses exclusively on network 
performance/scalability, and then goes on to praise or totally disqualify the 
OS's tested solely on this ground. The test doesn't look at crucial aspects 
like security, maintainability, documentation, correctness of design, 
etcetera, but will still not hesitate to draw very un-subtle conclusions.

- Ask those who are in the know for comments on the test conclusions before 
making the results public.
In this precise case, it would have prevented several stupid factual mistakes 
from being published (the OpenBSD installation problem and IPv6 
idiosyncrasies for instance.)

- Call into the test all relevant players, or at least represent different 
groups properly.
Much as I dislike it - I'm an open source adept as well - Windows is used as a 
server system on the public internet by many organisations. It should be 
compared with the other systems in this test, and be given a fair chance. I 
wouldn't have been surprised if it performed very well, seeing that the 
TCP/IP stack of modern Windows versions has been largely copied from 
FreeBSD..
Also I would have liked it if there were at least one proprietary Unix system 
(such as Solaris) in the test. Just for the sake of the comparison.

- Look closer into the reasons, backgrounds, pros and cons of faults that the 
tests find. Again, if the tester had done this, he would have found that some 
of the badness he found wasn't a design mistake, but a design decision based 
on healthily made trade-offs (security trade-offs in the case of, not 
surprisingly, OpenBSD.)

- Bert