BMERC benchmarks

The next-to-last host in the list, amdk6, is my vintage 1999 home computer, a 300MHz Linux PC with an AMD K6 Pentium-clone processor and 28MB of RAM. The last host, sigler, is a much newer (March 2001) PC with a 1200MHz AMD processor and 512MB of RAM.

For reference, here are miescher and vavilov before they were upgraded (i.e. the performance of the old boxes):

Results of testing the new vavilov and miescher are shown in the main table.

General comments

Timings were produced on an unloaded machine (to the extent possible), as they are intended to give an idea of the best realistic performance for that machine on that class of problem. The shorter tasks (specifically, new-stat and stat-test) were repeated twice, and the faster time (almost always the second one) was used.

The values given for each benchmark are of the form

    2.3 (63%)

    12.0u 6.0s 0:28 63% 0+0k 0+0io 0pf+0w

I use gamow as the standard mostly because it is the machine on my desk; since the speed value for this example is 2.3, that means it runs 2.3 times as fast on that machine as it does on gamow (which took 1:03 on this problem). It also helps that gamow is not a server machine, which improves the reliability and reproducibility of timings done there. Finally, picking a slow machine as a standard makes the speed multiples mostly greater than one, which make comparing them more intuitive.

The table still has a few blank spots; it has taken quite a while to fill it out even this far, especially since I feel no pressure. Some machines (e.g. pauling and gamow) are almost identical in configuration, and should produce almost identical results.

A note on I/O traffic

It is interesting to compare the times for delbrueck to those of the other machines in its class, feynman and mbcrrc. Since delbrueck is the file server, its speed multiples for the I/O-bound tasks are more in line with the speed multiple for 1bmt-cva. This implies that gamow is not actually I/O-bound for these tasks; delbrueck can supply gamow with data as fast as gamow can accept it (though the low processor utilization for gamow is puzzling). feynman and mbcrrc have 1bmt-cva multiples similar to delbrueck, but, although the data are ambiguous, appear to lose a factor of two when they need to rely heavily on delbrueck for file service.

There may also be some vendor-dependence in file I/O performance, in that Sun will tune their NFS client using Sun servers, and DEC will tune their  NFS client using DEC servers, so a DEC machine talking to a Sun server may experience a larger performance hit for an I/O-bound job than a Sun machine performing the same job. This might explain why the Alphas do proportionally less well than the Solaris boxes on this task (though data are still scant).

Explanation of benchmarks

new-stat

This directory contains 5000+ lines of C source code, so this task tests make & gcc.

    cd ~thread/code/stat/new-stat/
    uptime; make clean; time make install; uptime

The standard time for this task on gamow is 1:03. The elapsed times range from 7 seconds to over four minutes (the latter on jrsadler, on which gcc seems to run painfully slowly). The roundoff error can be more than 10% on the faster machines, since those times only have one significant digit.

stat-test

Runs the code compiled above ( mrf-envs, mrf-counts, and mrf-scores, plus the sing-envs.pl script), which do a little bit of floating point, more perl to check the results, and lots of I/O throughout.

    cd ~thread/code/stat/test
    uptime; make clean > /dev/null; time make check > foo.text; uptime

This takes 8:17 on gamow.

make-core

Runs the DSSP code and the perl scripts necessary to produce the test set of cores included on the needle tools distribution tar file. (May be sensitive to system/version differences in DSSP speed.)

    cd ~thread/code/stat/dist/test-cores
    uptime; make clean > /dev/null; time make; uptime

This takes 32:34 on gamow. Since the perl scripts use internal pipes, it runs markedly faster on machines with multiple processors (the Solaris boxes each have four, as I understand it). It could run even faster on such machines by using the parallel processing capability of GNU make, but I am not doing this (as far as I know).

1bmt-cva

This runs calculate-vv-all (mostly), which is compute- and floating-point intensive, plus some testing scripts, on a chain of 246 residues.

    cd ~thread/code/stat/test
    uptime; rm -f new-env_1bmtA.pair; time make cmp-1bmtA; uptime

The calibration run on gamow takes fully 1:09:01, so the resolution is quite good even for the fastest machines. Note how the processor utilization figures are all up in the high 90's. This is because calculate-vv-all does little I/O and (apparently) doesn't need to page.