Cluster Challenge 2008 – an adviser’s perspective

I did the Cluster Challenge again this year. Last year was fun, but this year was better – we won! Here’s the story:

We started Saturday morning in Bloomington. The travel went pretty
smooth and Guido picked us up at the airport in Austin. We went directly
to the Conference location. The time before that was very stressful
because the machine wasn’t working quite as nice as we would like to.
The biggest problem was that the we could not change the CPU frequency
of the new iDataplex system. However, we were able to change it on the
older version and saw significant gains. We benchmarked that we could
run 16 nodes during the challenge and use 12 of them for HPCC (while 4
were idle) with our power constraints (2×13 A). So we convinced IBM to
give us a pre-release BIOS update which was supposed to enable CPU
frequency scaling. And it looked good! We were able to reduce the CPU
clock to 2.0 GHz (as on the older systems). However, it was 4am and we
had to ship at 6, so we didn’t have the time to test more. But back to
Austin …

The guys from Dresden were already waiting for us because the organizers
did not allow them to unpack the cluster alone (it was supposed to be a
team effort). We unpacked our huge box and rolled our 900 pound cluster into
our booth.

Our Cluster

We spent the remaining day with installing the system and pimping (;-))
our booth. It went pretty well. Then we began to deploy our hardware and
boot it from USB to do some performance benchmarks.

Installing the fragile Fiber Myrinet equipment (we didn’t break anything!)

We started with HPCC and were shocked twice. Number one was that the CPU
scaling that costed us so many sleepless night did not seem to help. All
tools and /proc/cpuinfo showed 2.0 GHz – but the power consumption was
still as high as with 2.5 GHz. So we wrote a small RDTSC benchmark to
check the CPU frequency – it still ran at 2.5 GHz. The BIOS was lying to
us :-(.  The second shock was that HPL was twice as slow as it should
be. So much to the sleep …
Quite some time after midnight … still hacking on stuff. I’m trying to motivate (I am a good slave driver) our guys to go on.

The students tried to fix it … all night long. The conclusion was that
we had to drop our cool booting from USB idea due to the performance
loss. Later, it turned out that shared memory communication uses /tmp
(which was mounted via NFS) and was thus really slow (WTF!). Anyway, we
decided about one hour before the challenge started to fall back to
disks. This worked.

How high can one stack harddrives? Not too high actually ;). Man, this was hard to plug them back into the system.

The second problem was a tough one. The BIOS … lying to us. We were
finally able to get hold of an engineer from IBM. He tried hard but
couldn’t help us either. So the students had to make the hard decision
to run with two nodes less :-(.

In the meantime, me and Bob had fun while biking in order to power
laptops ;).

Bob Beck (UA) generating power on our fancy machine ;).

I was driving my laptop with the sandwiches I ate before :).

The Challenge finally starts

The challenge was about to start, the advisors couldn’t do anything
anymore, so we decided to get some fuel from the opening banquet for our
students in the nightshift ;).
Guido and me thinking about getting some good stuff for the students!

We finally found some good stuff on the showfloor *yay*.Advisor’s success!

Some of us were not totally up to speed all the time 😉 – It looks like somebody missed the start:

So the Challenge ran, and we had nothing to do (especially the advisers
who were just hanging around to feed and motivate the students). So we
did all kinds of weird things over night – and we had a bike ;).

I also started some coding during the challenge because I didn’t really
do anything but it was way to noisy to work on papers. I had to pose
inside the microsoft booth, while my laptop finished up some cool
things! Thanks to Erez for taking the picture at exactly the right time.

Some Linux-based “research” performed/finished inside the Microsoft booth.
Guido explains Vampir to the other teams on one of our three ultra-cool
41” displays (again, around midinght ;)). We had really nice speakers
at the challenge. Especially on Sunday, when all the others left, we
cranked them up and listened to the soundtrack of Black Hawk down. The
security guys seemed kind of confused to hear really loud base at 4am in
the morning ;).

Guido! Don’t help the “enemies” ;).

Youtube made it also on our display :). And nearly costed us a point by
disturbing the sound output of our power warning system. But Jens
realized it fortunately.

Watch yourself:  Achmed the Dead Terrorist

Oh, and there was this Novell penguin that spontaneously caught fire. I
guess this happens when experienced computer scientists spend two days
to install a completely retarded operating system (with InfiniBand – ask
me about details if you’re interested). I love Linux, but it’s a shame
that the abbreviation SLES has the word “Linux” in it. Debian or Ubuntu
is so much better! But apparently, SLES is better prepared for the
applications (clearly not for administration or software maintenance

Each booth was “armed” with at least one student during all the time.
Here are some images from after midnight.

The MIT booth – doesn’t it look more like Stonybrook?

The folks from Arizona State – they had a neat Cray – with Windows though. But it seems that it worked for them.

The guys from Colorado with Aspen systems (don’t ask them about their vendor partner).

The National Tsing Hua University – excellent people but their system was more of a jet engine than a cluster.

Our booth … note the image on the big screen ;).

The Alberta folks – last year’s champions. Darn good hackers!

Purdue with their SciCortex – they seemed rather annoyed all the time.

Our social corner: At 2am, most students didn’t have to do a lot (just
watching jobs). So they all gathered in front of our booth and played
cards :).

Two fluffy spectators were watching our oscilloscope animation during
the show-off on Thursday.

The team of judges, led by Jack Dongarra, talked to our students to
assess their abilities

After that, we won! We don’t have a picture of our fabulous win yet, but I’ll post it with some more links after I got it.

My first quotable citation

I actually came up with this a while back. I think now, after going through this process, I can say this:

“You don’t do research to get a Ph.D., you get a Ph.D. to do research.”

Also a variant with “shouldn’t” would be appropriate ;).

Or in German:

“Man forscht nicht um zu promovieren, sondern man promoviert um zu forschen.”

Die Variante mit “sollte” ist auch passend :).

You can also call me Dr. in Germany!

no worries, nobody has to ;). But since March 6th 2008, when the “Kultusministerkonferenz” (the federal secretary of education meeting) decided that American Ph.D.s from universities listed as Ph.D. granting research institutions in the Carnegie List (Indiana University is of course on that list!) can use the academic title “Dr.” in Germany without limitations. Wow, it took them until 2008 to realize that – amazing. It also says that I can’t use both at the same time (too bad ;)). Anyway, I guess that saves me many problems :).

I like the formulation:

“Da es verschiedentlich zu Anzeigen wegen angeblich missbraeuchlicher Fuehrung von Doktorgraden gekommen ist, wird die Fuehrung von auslaendischen Doktorgraden neu geregelt:
Inhaber des Doktorgrades “Doctor of Philosophy” Abk.: “Ph.D.” von Universitaeten der sog. Carnegie-Liste der Vereinigten Staaten von Amerika koennen anstelle der im Herkunftsland zugelassenen oder nachweislich allgemein Ueblichen Abkuerzung die Abkuerzung “Dr.” jeweils ohne fachlichen Zusatz und Herkunftsbezeichnung fuehren.”

I won’t translate it, but it starts with:”Because there were some people sued over the alleged use of doctoral degrees, we will rearrange the rules for carrying foreign doctoral degrees:”. I guess it was mostly a reaction to articles like this (German) or this (German/English) which describe an old Nazi-law that could probably scare some good researchers ;). Here’s the whole citation just for reference:

Non-European PhDs In Germany Find Use Of “Doktor” Verboten

By Craig Whitlock and Shannon Smiley
Washington Post Foreign Service
Friday, March 14, 2008; A01

BERLIN, March 13 Americans with PhDs beware: Telling people in Germany that you’re a doctor could land you in jail.
At least seven U.S. citizens working as researchers in Germany have faced criminal probes in recent months for using the title “Dr.” on their business cards, Web sites and resumes. They all hold doctoral degrees from elite universities back home.
Under a little-known Nazi-era law, only people who earn PhDs or medical degrees in Germany are allowed to use “Dr.” as a courtesy title.
The law was modified in 2001 to extend the privilege to degree-holders from any country in the European Union. But docs from the United States and anywhere else outside Europe are still forbidden to use the honorific. Violators can face a year behind bars.
Ian Thomas Baldwin, a Cornell-educated researcher at the Max Planck Institute for Chemical Ecology in Jena, has stopped calling himself “Dr.” ever since he was summoned for interrogation by police two months ago on suspicion of “title abuse.”
“Coming from the States, I had assumed that when you get a letter from the criminal police, you’ve either murdered someone or embezzled something or done something serious,” said Baldwin, a molecular ecologist. “It is absurd. I’s totally absurd.”
No one has questioned the legitimacy of his degree or whether he has the right to conduct research here. But going by “Dr.” is verboten. If he wants to refer to his doctorate, German law dictates that he identify himself as “Prof. Ian Thomas Baldwin, PhD, Cornell University.”
Baldwin confessed in a telephone interview that “there’s no question I’m guilty as charged.” But he hopes prosecutors will give him a break.
In his defense, he noted that the Max Planck Institute has always addressed him as “Prof. Dr. Baldwin” since it offered him a job a decade ago, and nobody warned him he might be in legal peril if he did likewise.
The proper use of honorifics is no small matter in Germany, a society given to formality where even longtime neighbors insist on addressing each other using their surnames. Those with advanced degrees like to show them off, and it is not uncommon to earn more than one. A male faculty member with two PhDs can fully expect to be called “Herr Professor Dr. Dr. Schmidt,” for example.
In effect, forcing Americans to forsake their titles amounts to a social demotion. “It’s an indication of the hierarchization of German society,” said Gary Smith, director of the American Academy in Berlin. “Germans are much more status-conscious about these things, and the status is real.”
Smith holds a doctorate from Boston University and has tempted fate by answering to “Dr. Smith” during the two decades he’s lived in Germany. He said he was told years ago that there is a legal way for foreign PhDs and MDs to register for permission to use the appellation, but he has never bothered.
“It wasn’t worth the trouble of doing anything about it,” he said. “It’s really an absurd situation in a globalized world.”
The German doctor rule has been in effect since the 1930s, but it has been only sporadically enforced in recent years.
That changed last fall, when an anonymous tipster filed a complaint with federal prosecutors against seven Americans at the prestigious Max Planck Society, which operates 80 scientific research institutes across Germany. Federal authorities forwarded the complaint to prosecutors and police in at least three states, who decided to take action.
Joerg Stolz, the chief prosecutor in the city of Jena, which is investigating Baldwin and another researcher at the Max Planck Institute there on suspicion of title abuse, said those two probes were “near closure.”
He said his office had recommended to a judge against filing charges. In that event, he said, the matter would be referred to the Cultural Ministry in the state of Thuringia, which could still decide whether a civil fine is warranted.
Detlef Baer, a spokesman for the ministry, said officials planned to drop both cases. “We spoke with the parties involved and determined they had no criminal intent,” he said. “They were given instructions as to how they can refer to their titles,” by citing the degree but not calling themselves doctors.
Another American investigated by police is an astrophysicist with a doctorate from Caltech and membership in the German Academy of Sciences.
The criminal investigations have alarmed higher education officials in Germany, where U.S. researchers are in high demand and treated as blue-chip recruits. Last week, state education ministers met in Berlin and recommended that the law be modified so anyone holding a doctorate or medical degree from America could be addressed as “Dr.” without running afoul of the police.
“This is a completely overdone, mad, absolutely ridiculous situation,” said Barbara Buchal-Hoever, head of Germany’s central office for foreign education. “We are talking about highly acclaimed researchers here. . . . The people who have pressed charges must be gripers or troublemakers who wanted to make a totally absurd point.”
Even if the proposal is adopted, however, it would extend the privilege only to people with degrees from about 200 U.S. universities accredited by the Carnegie Foundation for the Advancement of Teaching. Anyone with a PhD from Canada, Japan or the rest of the non-European world would still be excluded.
For now, the old law remains on the books. It is unclear when, or if, Germany’s state parliaments will change it.
So the next time Dr. Condoleezza Rice (PhD, University of Denver) or even German-born Dr. Henry Kissinger (PhD, Harvard) pay a visit to Berlin, they may want to stick with the title “secretary of state.”

Living in the UITS datacenter

Timo (my visiting student from Germany) and me are pretty much living in the UITS data-center (Wrubel building) these days. It’s a rather nice serverroom and we’re having our cluster directly next to the Data Capacitor and Big Red.

It was nice the last days, we stayed overnight during the weekend (more than 16 hours a day where others celebrated Halloween) and some nights during last week. Prof. Lumsdaine was so generous to buy us pizza at night.Thanks a lot!
The fun fact is the some caperpillar ran over a huge power transformer yesterday and cut off a part of the power supply to the building.The immediate results was that they had to switch off the air conditioning and some systems (Big Red was among them). It was rather funny to be in a really big data-center without air conditioning. It got hotter literally every hour, I guess it reached about 30-35 degrees at the end (we were sweating while just sitting there). The biggest problem was that we did power measurements but our system used 0.3-0.4 A more power than usual :-(. And we don’t know if it is linear :-/. We still ran some benchmarks. Anyway, the power and cold air came back around 2am. The IU physical plant people were really quick in getting a new transformer from Cincinnatti by truck (in less than 6 hours).

We then tried to get some food at Wednesday night 2:30am … man, all pizza places were closed. So we decided to drive to Taco Bell and have a “fourth meal”. So paked next to it and tried to walk into the restaurant – but it was locked. Ok, the drive-in was still open (how weird). Shortly after we left, the police stopped us and asked us if we had gasoline in the car (what a *really stupid questions* – no, our car is running with hydrogen … man, those people). Anyway, we realized that they were searching for a firebug and the Taco Bell people told them that we were suspicious. WTF! Again, man, those people.

Cluster Challenge is really fun again. I’m happy that Guido and Jupp are here to help us. It’s getting really close but our system is really nice ;).