Welcome to part two of the Align InfiniBand blog series. In this post, Rodney Willis, Align’s VP of Sales and Sourcing, completes a deep dive into the various types of cabling media and termination types used in InfiniBand networks. He also discusses the next generation of fiber connections.If you missed part one, Demystifying the InfiniBand Versus Ethernet Debate, you can catch up here.
To further understand the use cases and applications for InfiniBand networks, we’ll start off
Cable termination refers to the way in which the ends of the cables used for transmitting data are finished or connected. It's how the cable connects to the equipment or interfaces at either end of the communication link.
Below are some examples of common termination types and the transmission media that supports them:
MPO (Multi-Fiber Push On) ((Tissue)) / MTP (Multi-Fiber Termination Push-on)((Kleenex)): This is a confusion in the marketplace. In essence, MPO and MTP are the same form factor. They are functionally similar and mainly differ by the manufacturer and their tolerances and specifications. They are the de facto standard for InfiniBand connectivity and trunking ethernet backbones for fiber optic connections.
LC Low Loss Connector: Developed by Lucent, many people believe that this stood for Lucent Connector, but it’s actually just referring to a low loss connector as it was working through the standards committees to be ratified as a connector type for multi-mode and single-mode fiber terminations. Typically associated with Ethernet infrastructure.
RJ45: The “Registered Jack“ was developed as part of the Bell Labs development systems back in the 1970s. The “45” in the name references the listing number for the specification and is an 8 Pin / 8 Conductor Ethernet connector. The transmission media for this connection style is Shielded (STP), Screened (ScTP), but more predominantly Unshielded Twisted Pair (UTP) copper cabling. The transmission media for the RJ45 Solution would be rated in “Categories” Cat 6, Cat 6A, Cat 7 and the mythical Cat 8 cable that manufacturers have promoted in a theoretical concept for several years now.
Direct attach Twinax cables: QSFP (+) / QDR interfaced cables that are designed with a specific length of cable and intended for shorter distances. The twinax cable is connectorized directly to the QSFP / QDR interfaces and intended to bypass a standard interface and plug directly into a device. Typically, these are not ideal for inter-cabinet connections due to routing and MAC work.
Active optical cables (AOCs) are a type of Hybrid cable lengths that are terminated directly on each side with the optical transceiver directly terminated to the fiber optic transmission media. Typically, on both sides but more recently as a hybrid termination and the AOC is only single ended with an a host of options on the other .
The way cables are terminated depends on the pair count / strand count, and the interface they are connecting to. The hybrid cables that were mentioned earlier are active optical cables and direct attach cables where you are bypassing the industry standard part of interfacing and connecting equipment. For example, in an active optical cable, if the transceiver company can focus on knowing what their optics output is, they could potentially use a lower grade of fiber. The downfall of this approach is you don't have standards-based interoperability, and you have to replace the cable and the transceiver all in one if something happened to either component.
Additionally, what terminations you choose is determined by whether you are deploying a structured cabling system or a point-to-point cabling infrastructure. Again, I believe structured cabling gives the most flexibility. Point-to-point is definitely the most cost-effective solution but based on the evolution of the GPU and the infrastructure needed to support it, it might not be the best long-term solution. As we talked about in the last blog, the lines are being blurred from InfiniBand to Ethernet or converge to Ethernet.
In a structured cabling approach, you have more connectivity options and ways to future proof your infrastructure to support your AI clusters and InfiniBand (or whatever transmission protocol your current and next generation of equipment supports).
The ongoing evolution in fiber connectivity mirrors past battles between competing connector technologies, reminiscent of the LC versus Mt RJ showdown. With contenders like Senko's SN-MT and US Conec’s MMC boasting ultra-compact 16-fiber connectors, the industry is poised for transformative change. As demands for high-density fiber solutions surge, particularly within AI-driven infrastructures, the race for more efficient, higher-capacity fiber connections intensifies, reshaping the landscape of networking technologies.
The selection of transmission media is extremely important when setting up a network and connecting devices. It depends on factors such as the:
Based on a scenario where equipment is 38 meters apart, I would select an infrastructure of OM4 fiber with MPO 8 angle polished MPO connections between the OSFP 400G transmissions. Let’s consider some specifications to explain this:
Just a few years ago, what we deemed cutting-edge quickly became outdated. With the staggering pace of innovation pushing the boundaries of GPU technology, we find ourselves navigating multi-generational leaps within a very short time frame. In this kind of environment, the traditional approach of relying on set-defined cabling infrastructure proves impractical and cost-ineffective. At its core, structured cabling involves organizing the network into a series of manageable and easily identifiable components, including cables, connectors, and hardware. Unlike ad-hoc cabling setups, which often result in a tangled mess of wires and disparate systems, structured cabling adheres to industry-established standards and best practices. This means employing uniform cable types, termination methods, and labeling schemes throughout the network, regardless of its size or complexity. By adopting a structured approach, companies benefit from simplified troubleshooting, streamlined maintenance, and greater scalability, and overall future-readiness.
Before moving on to cables and terminations specifically, I’d like to call out that terms like OSFP, QSFP or CX4 are sometimes used interchangeably with "cables," leading to misunderstandings. These are the optics and transceivers that the transmission cables and terminated connectors interface with…not the cables themselves.
Several members of the Align team attended the NVIDIA GTC conference a few weeks ago to get a look at the next generation Blackwell chip and the new 120kW+ DGX cabinet. The intra cabinet cabling in the new DGX threw us a curveball. When speaking to an engineer at their booth, the engineer indicated that the 2 miles of new intra cabinet cabling was direct soldered twinax cable. I’d say that’s a red flag. The break/fix and MTTR could be problematic, and I don’t think anyone would want to live solder in the back of a ¼ million dollar + server.
Learn more about our Data Center Solutions by clicking here, or contact us today to get in touch with one of our team members.
And stay tuned for more on this topic as we continue with our InfiniBand Series over the coming weeks.