Welcome to part two of the Align InfiniBand blog series. In this post, Rodney Willis, Align’s VP of Sales and Sourcing, completes a deep dive into the various types of cabling media and termination types used in InfiniBand networks. He also discusses the next generation of fiber connections.If you missed part one, Demystifying the InfiniBand Versus Ethernet Debate, you can catch up here.
To further understand the use cases and applications for InfiniBand networks, we’ll start off
with a detailed description of components and termination, considerations for selecting transmission media, and begin discussing the importance of structured cabling when establishing these networks.
Cable Termination in Networks
Cable termination refers to the way in which the ends of the cables used for transmitting data are finished or connected. It's how the cable connects to the equipment or interfaces at either end of the communication link.
Below are some examples of common termination types and the transmission media that supports them:
MPO (Multi-Fiber Push On) ((Tissue)) / MTP (Multi-Fiber Termination Push-on)((Kleenex)): This is a confusion in the marketplace. In essence, MPO and MTP are the same form factor. They are functionally similar and mainly differ by the manufacturer and their tolerances and specifications. They are the de facto standard for InfiniBand connectivity and trunking ethernet backbones for fiber optic connections.
LC Low Loss Connector: Developed by Lucent, many people believe that this stood for Lucent Connector, but it’s actually just referring to a low loss connector as it was working through the standards committees to be ratified as a connector type for multi-mode and single-mode fiber terminations. Typically associated with Ethernet infrastructure.
RJ45: The “Registered Jack“ was developed as part of the Bell Labs development systems back in the 1970s. The “45” in the name references the listing number for the specification and is an 8 Pin / 8 Conductor Ethernet connector. The transmission media for this connection style is Shielded (STP), Screened (ScTP), but more predominantly Unshielded Twisted Pair (UTP) copper cabling. The transmission media for the RJ45 Solution would be rated in “Categories” Cat 6, Cat 6A, Cat 7 and the mythical Cat 8 cable that manufacturers have promoted in a theoretical concept for several years now.
Hybrid Connections
Direct attach Twinax cables: QSFP (+) / QDR interfaced cables that are designed with a specific length of cable and intended for shorter distances. The twinax cable is connectorized directly to the QSFP / QDR interfaces and intended to bypass a standard interface and plug directly into a device. Typically, these are not ideal for inter-cabinet connections due to routing and MAC work.
Active Optical Cables
Active optical cables (AOCs) are a type of Hybrid cable lengths that are terminated directly on each side with the optical transceiver directly terminated to the fiber optic transmission media. Typically, on both sides but more recently as a hybrid termination and the AOC is only single ended with an a host of options on the other .
Terminations / Interfacing
The way cables are terminated depends on the pair count / strand count, and the interface they are connecting to. The hybrid cables that were mentioned earlier are active optical cables and direct attach cables where you are bypassing the industry standard part of interfacing and connecting equipment. For example, in an active optical cable, if the transceiver company can focus on knowing what their optics output is, they could potentially use a lower grade of fiber. The downfall of this approach is you don't have standards-based interoperability, and you have to replace the cable and the transceiver all in one if something happened to either component.
Additionally, what terminations you choose is determined by whether you are deploying a structured cabling system or a point-to-point cabling infrastructure. Again, I believe structured cabling gives the most flexibility. Point-to-point is definitely the most cost-effective solution but based on the evolution of the GPU and the infrastructure needed to support it, it might not be the best long-term solution. As we talked about in the last blog, the lines are being blurred from InfiniBand to Ethernet or converge to Ethernet.
In a structured cabling approach, you have more connectivity options and ways to future proof your infrastructure to support your AI clusters and InfiniBand (or whatever transmission protocol your current and next generation of equipment supports).
Next Generation of Fiber Connections
The ongoing evolution in fiber connectivity mirrors past battles between competing connector technologies, reminiscent of the LC versus Mt RJ showdown. With contenders like Senko's SN-MT and US Conec’s MMC boasting ultra-compact 16-fiber connectors, the industry is poised for transformative change. As demands for high-density fiber solutions surge, particularly within AI-driven infrastructures, the race for more efficient, higher-capacity fiber connections intensifies, reshaping the landscape of networking technologies.
Selecting the Transmission media
The selection of transmission media is extremely important when setting up a network and connecting devices. It depends on factors such as the:
- Distance between devices
- Device interface
- Specific requirements of transmission protocol that is being transported over transmission media
Selection— A Small Example
Based on a scenario where equipment is 38 meters apart, I would select an infrastructure of OM4 fiber with MPO 8 angle polished MPO connections between the OSFP 400G transmissions. Let’s consider some specifications to explain this:
- A selected “400” gig InfiniBand OSFP connection operates at the 850nm wavelength, the connector style is Angled MPO-12 but only uses 8 of the 12 fiber lanes (4 lanes of transmit and 4 lanes of receive)
- The VCSEL transmission is 4 fibers of 106.25G each direction and modulated together for a max data rate of 425Gbps
- The two predominate grades of multi-mode fiber that could transmit over the 850 nm wavelength are OM3 and OM4
- OM3 is rated for 30 meters and OM4 is rated at 50 meters
Structured Cabling
Just a few years ago, what we deemed cutting-edge quickly became outdated. With the staggering pace of innovation pushing the boundaries of GPU technology, we find ourselves navigating multi-generational leaps within a very short time frame. In this kind of environment, the traditional approach of relying on set-defined cabling infrastructure proves impractical and cost-ineffective. At its core, structured cabling involves organizing the network into a series of manageable and easily identifiable components, including cables, connectors, and hardware. Unlike ad-hoc cabling setups, which often result in a tangled mess of wires and disparate systems, structured cabling adheres to industry-established standards and best practices. This means employing uniform cable types, termination methods, and labeling schemes throughout the network, regardless of its size or complexity. By adopting a structured approach, companies benefit from simplified troubleshooting, streamlined maintenance, and greater scalability, and overall future-readiness.
Before moving on to cables and terminations specifically, I’d like to call out that terms like OSFP, QSFP or CX4 are sometimes used interchangeably with "cables," leading to misunderstandings. These are the optics and transceivers that the transmission cables and terminated connectors interface with…not the cables themselves.
An Anecdote From the Ground Floor:
Several members of the Align team attended the NVIDIA GTC conference a few weeks ago to get a look at the next generation Blackwell chip and the new 120kW+ DGX cabinet. The intra cabinet cabling in the new DGX threw us a curveball. When speaking to an engineer at their booth, the engineer indicated that the 2 miles of new intra cabinet cabling was direct soldered twinax cable. I’d say that’s a red flag. The break/fix and MTTR could be problematic, and I don’t think anyone would want to live solder in the back of a ¼ million dollar + server.
Speak to an Align Data Center Expert:
Learn more about our Data Center Solutions by clicking here, or contact us today to get in touch with one of our team members.
And stay tuned for more on this topic as we continue with our InfiniBand Series over the coming weeks.
