Cactus Gets Highest Bandwidth In SC2001 Challenge

From: LBNL In-The-Loop Newsletter
Author: Jon Bashor

Lab Team Surpasses Own Expectations in Winning SC2001 Bandwidth Challenge

When a Berkeley Lab-led team won the High-Performance Bandwidth Challenge at the SC2001 conference last week for the second year in a row, their sustained network saturation rate of 3.3 gigabits per second exceeded even the team's expectations. In their entry for the competition, team leader John Shalf said they expected to move 2.5 gigabits, completely saturating the specially installed OC-48 connection between NERSC and the conference site in Denver. ESnet staffers made an extra effort to get that link up and running in time for SC2001.

Team leader John Shalf accepting the High-Performance Bandwidth Challenge award at SC2001 in Denver

"Members of the ESnet staff really went out of their way to get the OC-48 link up and running to make our participation in the High-Performance Bandwidth Challenge possible," said Bill Kramer, head of NERSC's High Performance Computing Department and chair of the bandwidth competition. "In addition to the challenge team members, ESnet deserves a lot of credit for helping the Lab team win."

Qwest Telecommunications, Level 3 and Juniper Networks also contributed expertise and equipment to the effort.

The team was also able to take advantage of two other OC-12 network connections, one ESnet connection from NERSC and the other from the National Center for Supercomputing Applications in Illinois via the Abilene network, to push their data transfer to the 3.3 Gbits rate. The group ran a simulation of a grazing collision of two black holes using the Cactus simulation code developed by collaborators at the Albert Einstein Institute in Germany on NERSC's 5 teraflop/s IBM SP-2 system. The data from that running simulation was then sent realtime via high-performance networks to the Denver where it was volume-rendered in parallel using the Visapult application running on a cluster of PCs in the LBNL booth on the SC2001 show floor. The application provided highly interactive visualization and computational steering of a production-scale simulation code over a wide area network.

The team also had a new piece of equipment to help them out - a 10 Gigabit Ethernet switch built by Force 10 Networks. SC2001 marked the public debut of the new switch, one of which was in the Lab booth and the second was in the conference network operations center. John Christman of LBLnet facilitated the use of the switches from Force 10. The team used the switch to achieve a high bandwidth connection between various sites. "The 10 Gigabit switch was one of the few trouble-free components of this entire network-distributed application," said John Shalf. "Despite the performance of the Visapult/Cactus application, we never got close to stressing the Force 10 switch."

"High Performance Computing and 10 Gigabit Ethernet are a natural fit, especially for distributed supercomputing clusters that require significant bandwidth between the distributed sites," said Steve Mullaney, vice president of marketing for the Milpitas-based Force 10. "And as supercomputing nodes move to Gigabit Ethernet speeds, 10 Gigabit Ethernet is needed to aggregate the Gigabit Ethernet connections in scalable, non-blocking networks."

For more information about the winners of the High-Performance Bandwidth Challenge, go to http://www.lbl.gov/Computing-Sciences/Archive/headlines111601.html