Innovatives Supercomputing in Deutschland
inSiDE • Vol. 9 No. 1 • Spring 2011
current edition
about inSiDE
index  index prev  prev next  next

New 100 Gigabit/s Ethernet Technology successfully tested between FZJ and KIT

In a joint effort, the German Research Network Provider DFN, the Jülich Supercomputing Centre (JSC) at the Forschungszentrum Jülich (FZJ) and the Karlsruhe Institute of Technology (KIT) as well as the industrial partners Cisco Systems, Huawei Technology and GasLINE tested the newly developed 100 Gigabit/s Ethernet technology on a 447 km connection between JSC and KIT.

Figure 1: Huawei DWDM-transmission system with 100GE wavelength conversion board

Facing growing demand for communication bandwidth e.g. in the area of Telecommunication Provider Backbones or Supercomputer I/O, in summer 2010 the IETF and IEEE finalized the IEEE802.3ba standard for 100 GBit/s Ethernet. This standard especially addresses today’s limiting factors for high bandwidth communication: constraints concerning the number of parallel links that can be combined into a single logical link (IEEE802.3ad link aggregation) and the number of wavelengths that can be transmitted in parallel over a single fibre with DWDM (dense wavelength division multiplexing).

Figure 2: Cisco CRS-3 and Huawei NE5000

After several weeks of preparation and the installation of multiple racks of network equipment, partly still in beta-status, the tests between JSC and KIT started in autumn 2010 via a 447 km fibre path with 6 optical regenerators provided by GasLINE. The optical layer consisted of a Huawei DWDM transmission system with 100 Gbit/s optical transceivers (so called 100GBase-LR4 CFPs) at the customer side interface. While during the setup phase some incompatibility issues between different CFP-vendors were found on the optical layer, these problems could be solved by CFP exchanges and rearrangements. During the 3 weeks lasting first test-phase a jointly administered carrier-class router Cisco CRS-3 per site aggregated up to twelve 10 GBit/s Ethernet-streams generated by test nodes. The tests with UDP- and TCP-based traffic patterns were accompanied by continuous monitoring of traffic statistics and the measurement of packet loss, delay and variation via additional dedicated measuring systems. We could show that a sustained throughput of 99.6 GBit/s was achievable via this long range 100 GBit/s Ethernet connection. In a second, 10 days lasting test-phase, a similar test scenario was set up with Huawei NE5000/NE40E routers. With intensive support of Huawei engineers the Huawei routers were able to drive the 100 GBit/s Ethernet connection close to 100% utilization, too.

Figure 3: Linux- based test nodes at the JSC

As a result of months of preparation and testing it turned out that the new 100 GBit/s Ethernet standard is already well implemented and supported by network components (routers, DWDM transmission systems) of major vendors and ready to be deployed in real-world production networks.

Figure 4: UDP traffic forces multiple TCP-streams on the 100GE-link to adapt and slowdown

• Olaf Mextorf
Jülich Supercomputing Centre

top  top