InfiniBand Network and Fat-tree Topology in HPC/AI: Accelerating High Performance Computing and Artificial Intelligence

2024-09-12 10:45

High-performance computing (HPC)/AI covers advanced computing with parallel processing, covering a wide range of scientific fields, such as meteorology, astronomy and seismology. It needs to simulate the motion of the atmosphere and ocean. In the field of artificial intelligence, it needs deep learning, large-scale data analysis and machine learning. High-performance simulation needs an efficient computing platform, and the execution time of a given simulation depends on many factors, such as the number of CPU/GPU cores and their utilization rate, as well as interconnection performance, efficiency and efficiency. An efficient HPC system usually consists of thousands, tens of thousands or even hundreds of thousands of computing nodes, each of which is equipped with high-speed processors, large-capacity memory and high-speed network connections.network topologyMany HPC networks can be built with IB, and InfiniBand structure has several common topologies. Some of these topologies are listed below:Fat tree: multiple trees. This is a popular topology.2D mesh: Each node is connected to the other four nodes; Positive, negative, x-axis and y-axis3D mesh: Each node is connected to the other six nodes; X, y and z axes are positive and negative.2D/3D Torus: The X, Y and Z ends of a 2D/3D mashup are "encircled" and connected to the first node.Dragonfly+: Each spine switch should be connected to all peer Spines in all other groups.1.1.Fat TreeFat-tree topology is a multi-tree network, which is widely used in HPC/AI. The widely used topology in HPC /AI cluster is the topology using fat-tree. When configured as a non-blocking network, this topology can usually achieve performance on a large scale. If the network convergence ratio can be tolerated, the cluster can also be configured as a blocking configuration.Fat-tree realizes the technology of non-blocking network, which can use a large number of low-performance switches to build a large-scale non-blocking network. fat-tree cluster usually uses the same bandwidth for all links, and in most cases, it uses the same number of ports in all switches.1.2 rules for building a Fat-Tree clusterThe following rules must be observed when designing Fat-tree cluster:Non-blocking clusters must be balanced. The same number of links must connect the Level-2 (L2) switch to each Level-1 (L1) switch. Whether convergence ratio can be adopted depends on HPC application and network requirements.If the L2 switch is a directional switch (that is, a switch with a Leaf board and a spin board), all links from the L1 switch to the L2 switch must be evenly distributed between the Leaf-spine. For example, if six links are running between L1 and L2 switches, they can be assigned to the Leaf board in the form of 1:1:1:1:1:1, 2:2:2, 3:3 or 6. It cannot be mixed, such as 4:2, 5:1.Do not create a route that must traverse the tree up, back, and then up again. This creates a situation called Credit Loops, which can be manifested as traffic deadlock in the cluster. Generally speaking, there is no way to avoid Credit Loops. Any fat tree with multiple directors and edge switches has physical cycles, which can be avoided by using routing algorithms (such as up-down).Try to always use the 32(NDR)/40(HDR) port switch as L1 and the steering switch as L2. If this principle cannot be realized, please consult Wang Derui (Tel: 025-86595105 15895983233 Website: http://www.njwdr.com/) to ensure that the cluster being designed does not include Credit Loops.1.3 Topology exampleCLOS-3 topology (non-blocking)1.3.1 80-node fat tree (HDR)Use 1U QM8700 switch.

image.png


1.3.2 800-node fat tree (HDR)Use 1U QM8700 switch or CS8500 modular switch.

image.png


1.3.3 1024 Node (NDR)

image.png

1.4 Performance calculationThe formula for calculating node performance in floating-point operations per second (FLOPS) is as follows:Node performance in FLOPS = (CPU speed in Hz) x (number of CPU cores) x(CPU instructions per cycle) x (number of CPUs per node)For example, for an Intel dual CPU server based on Intel E5-2690(2.9GHz 8-core) CPU:2.9 x 8 x 8 x 2 = 371.2 GFLOPS (per server).Note: The number of instructions per cycle of E5-2600 series CPU is equal to 8.To calculate the cluster performance, multiply the number of results by the number of nodes in the HPC system to get the theoretical peak. A 72-node fat tree cluster (using 6 switches) has:371.2GFLOPS x 72 (node) = 26,726GFLOPS = ~27TFLOPSA 648-node fat tree cluster (using 54 switches) has:371.2GFLOPS x 648 (node) = 240,537GFLOPS = ~241TFLOPSFor a fat tree with more than 648 nodes, HPC cluster must have at least 3 levels of hierarchy. About including GPUThe actual performance of a cluster depends on the cluster interconnection. On average, using 1 Gb Ethernet (GbE) connection will reduce cluster performance by 50%. With 10GbE, you can expect a 30% performance degradation. However, InfiniBand interconnection produces 90% system efficiency; Only 10% performance loss. See www.top500.org for more information.

image.png

Note: InfiniBand is the main interconnection technology in HPC market. InfiniBand has many features that make it an ideal choice for HPC, including:Low latency and high throughputRemote direct memory access (RDMA)Planar Layer 2 that can be extended to thousands of endpointscentralization of managementmultipathSupport multiple topologies
share
WeChat
Sina Weibo
QQ
QQ space
Douban
Baidu Post Bar
解决方案
Prev Software-defined, hardware-accelerated programmable InfiniBand NDR network empowers E-class AI and HPC platforms.
Next NVIDIA Magnum IO GPUDirect Storage overview guide