This morning the PCI Special Interest Group (PCI-SIG) is releasing the much-awaited final (1.0) specification for PCI Express 6.0. The next generation of the ubiquitous bus is once again doubling the data rate of a PCIe lane, bringing it to 8GB/second in each direction – and far, far higher for multi-lane configurations. With the final version of the specification now sorted and approved, the group expects the first commercial hardware to hit the market in 12-18 months, which in practice means it should start showing up in servers in 2023.
First announced in the summer of 2019, PCI Express 6.0 is, as the name implies, the immediate follow-up to the current-generation PCIe 5.0 specification. Having made it their goal to keep doubling PCIe bandwidth roughly every 3 years, the PCI-SIG almost immediately set about work on PCIe 6.0 once the 5.0 specification was completed, looking at ways to once again double the bandwidth of PCIe. The product of those development efforts is the new PCIe 6.0 spec, and while the group has missed their original goal of a late 2021 release by mere weeks, today they are announcing that the specification has been finalized and is being released to the group’s members.
As always, the creation of an even faster version of PCIe technology has been driven by the insatiable bandwidth needs of the industry. The amount of data being moved by graphics cards, accelerators, network cards, SSDs, and other PCIe devices only continues to increase, and thus so must bus speeds to keep these devices fed. As with past versions of the standard, the immediate demand for the faster specification comes from server operators, whom are already regularly using large amounts of high-speed hardware. But in due time the technology should filter down to consumer devices (i.e. PCs) as well.
By doubling the speed of a PCIe link, PCIe 6.0 is an across-the-board doubling of bandwidth rates. X1 links move from 4GB/second/direction to 8GB/second/direction, and that scales all the way up to 128GB/second/direction for a full x16 link. For devices that are already suturing a link of a given width, the extra bandwidth represents a significant increase in bus limits; meanwhile for devices that aren’t yet saturating a link, PCIe 6.0 offers an opportunity to reduce the width of a link, maintaining the same bandwidth while bringing down hardware costs.
|PCI Express Bandwidth
(Full Duplex: GB/second/direction)
PCI Express was first launched in 2003, and today’s 6.0 release essentially marks the third major revision of the technology. Whereas PCIe 4.0 and 5.0 were “merely” extensions to earlier signaling methods – specifically, continuing to use PCIe 3.0’s 128b/130b signaling with NRZ – PCIe 6.0 undertakes a more significant overhaul, arguably the largest in the history of the standard.
In order to pull of another bandwidth doubling, the PCI-SIG has upended the signaling technology entirely, moving from the Non-Return-to-Zero (NRZ) tech used since the beginning, and to Pulse-Amplitude Modulation 4 (PAM4).
Alongside PAM4 and FEC, the final major technological addition to PCIe 6.0 is its FLow control unIT (FLIT) encoding method. Not to be confused with PAM4, which is at the physical layer, FLIT encoding is employed at the logical level to break up data into fixed-size packets. It’s by moving the logical layer to fixed size packets that PCIe 6.0 is able to implement FEC and other error correction methods, as these methods require said fixed-size packets. FLIT encoding itself is not a new technology, but like PAM4, is essentially being borrowed from the realm of high-speed networking, where it’s already used. And, according to the PCI-SIG, it’s one of the most important pieces of the specification, as it’s the key piece to enabling (continued) low-latency operation of PCIe with FEC, as well as allowing for very minimal overhead. All told, PCI-SIG considers PCIe 6.0 encoding to be a 1b/1b encoding method, as there’s no overhead in the data encoding itself (there is however overhead in the form of additional FEC/CRC packets).
As it’s more of an enabling piece than a feature of the specification, FLIT encoding should be fairly invisible to users. However, it’s important to note that the PCI-SIG considered it important/useful enough that FLIT encoding is also being backported in a sense to lower link rates; once FLIT is enabled on a link, a link will remain in FLIT mode at all times, even if the link rate is negotiated down. So, for example, if a PCIe 6.0 graphics card were to drop from a 64 GT/s (PCIe 6.0) rate to a 2.5GT/s (PCIe 1.x) rate to save power at idle, the link itself will still be operating in FLIT mode, rather than going back to a full PCIe 1.x style link. This both simplifies the design of the spec (not having to renegotiate connections beyond the link rate) and allows all link rates to benefit from the low latency and low overhead of FLIT.
As always, PCIe 6.0 is backwards compatible with earlier specifications; so older devices will work in newer hosts, and newer devices will work in older hosts. As well, the current forms of connectors remain supported, including the ubiquitous PCIe card edge connector. So while support for the specification will need to be built into newer generations of devices, it should be a relatively straightforward transition, just like previous generations of the technology.
Unfortunately, the PCI-SIG hasn’t been able to give us much in the way of guidance on what this means for implementations, particularly in consumer systems – the group just makes the standard, it’s up to hardware vendors to implement it. Because the switch to PAM4 means that the amount of signal loss for a given trace length hasn’t gone up, conceptually, placing PCIe 6.0 slots should be about as flexible as placing PCIe 5.0 slots. That said, we’re going to have to wait and see what AMD and Intel devise over the next few years. Being able to do something, and being able to do it on a consumer hardware budget are not always the same thing.
Wrapping things up, with the PCIe 6.0 specification finally completed, the PCI-SIG tells us that, based on previous adoption timelines, we should start seeing PCIe 6.0 compliant hardware hit the market in 12-18 months. In practice this means that we should see the first server gear next year, and then perhaps another year or two for consumer gear.