Testing PCIe 4.0

It's been over a year since the first consumer CPUs and SSDs supporting PCIe 4.0 hit the market, so we're a bit overdue for a testbed upgrade. Our Skylake system was adequate for even the fastest PCIe gen3 drives, but is finally a serious bottleneck.

We have years of archived results from the old testbed, which are still relevant to the vast majority of SSDs and computers out there that do not yet support PCIe gen4. We're not ready to throw out all that work quite yet; we will still be adding new test results measured on the old system until PCIe gen4 support is more widespread, or my office gets too crowded with computers—whichever happens first. (Side note: some rackmount cases for all these test systems would be greatly appreciated.)

AnandTech 2017-2020 Skylake Consumer SSD Testbed
CPU Intel Xeon E3 1240 v5
Motherboard ASRock Fatal1ty E3V5 Performance Gaming/OC
Chipset Intel C232
Memory 4x 8GB G.SKILL Ripjaws DDR4-2400 CL15
Software Windows 10 x64, version 1709
Linux kernel version 4.14, fio version 3.6
Spectre/Meltdown microcode and OS patches current as of May 2018

Since introducing the Skylake SSD testbed in 2017, we have made few changes to our testing configurations and procedures. In December 2017, we started using a Quarch XLC programmable power module (PPM), providing far more detailed and accurate power measurements than our old multimeter setup. In May 2019, we upgraded to a Quarch HD PPM, which can automatically compensate for voltage drop along the power cable to the drive. This allowed us to more directly measure M.2 PCIe SSD power: these drives can pull well over 2A from the 3.3V supply which can easily lead to more than the 5% supply voltage drop that drives are supposed to tolerate. At the same time, we introduced a new set of idle power measurements conducted on a newer Coffee Lake system. This is our first (and for the moment, only) SSD testbed that is capable of using the full range of PCIe power management features without crashing or other bugs. This allowed us to start reporting idle power levels for typical desktop and best-case laptop configurations.

Coffee Lake SSD Testbed for Idle Power
CPU Intel Core i7-8700K
Motherboard Gigabyte Aorus H370 Gaming 3 WiFi
Memory 2x 8GB Kingston DDR4-2666

On the software side, the disclosure of the Meltdown and Spectre CPU vulnerabilities at the beginning of 2018 led to numerous mitigations that affected overall system performance. The most severe effects were to system call overhead, which has a measurable impact on high-IOPS synthetic benchmarks. In May 2018, after the dust started to settle from the first round of vulnerability disclosures, we updated the firmware, microcode and operating systems on our testbed and took the opportunity to slightly tweak some of our synthetic benchmarks. Our pre-Spectre results are archived in the SSD 2017 section of our Bench database while the current post-Spectre results are in the SSD 2018 section. Of course, since May 2018 there have been many further related CPU security vulnerabilities found, and many changes to the mitigation techniques. Our SSD testing has not been tracking those software and microcode updates to avoid again invalidating previous scores. However, our new gen4-capable Ryzen test system is fully up to date with the latest firmware, microcode and OS versions.

AnandTech Ryzen PCIe 4.0 Consumer SSD Testbed
CPU AMD Ryzen 5 3600X
Motherboard ASRock B550 Pro
Memory 2x 16GB Mushkin DDR4-3600
Software Linux kernel version 5.8, fio version 3.23

Our new PCIe 4 test system uses an AMD Ryzen 5 3600X processor and an ASRock B550 motherboard. This provides PCIe 4 lanes from the CPU but not from the chipset. Whenever possible, we test NVMe SSDs with CPU-provided PCIe lanes rather than going through the chipset, so the lack of PCIe gen4 from the chipset isn't an issue. (We had a similar situation back when we were using a Haswell system that supported gen3 on the CPU lanes but only gen2 on the chipset.) Going with B550 instead of X570 also avoids the potential noise of a chipset fan. The DDR4-3600 is a big jump compared to our previous testbed, but is a fairly typical speed for current desktop builds and is a reasonable overclock. We're using the stock Wraith Spire 2 cooler; our current SSD tests are mostly single-threaded, so there's no need for a bigger heatsink.

For now, we are still using the same test scripts to generate the same workloads as on our older Skylake testbed. We haven't tried to control for all possible factors that could lead to different scores between the two testbeds. For this review, we have re-tested several drives on the new testbed to illustrate the scale of these effects. In future reviews, we will be rolling out new synthetic benchmarks that will not be directly comparable to the tests in this review and past reviews. Several of our older benchmarks do a poor job of capturing the behavior of the increasingly common QLC SSDs, but that's not important for today's review. The performance differences between new and old testbeds should be minor, except where the CPU speed is a bottleneck. This mostly happens when testing random IO at high queue depths.

More important for today is the fact that our old benchmarks only test queue depths up to 32 (the limit for SATA drives), and that's not always enough to use the full theoretical performance of a high-end NVMe drive—especially since our old tests only use one CPU core to stress the SSD. We'll be introducing a few new tests to better show these theoretical limits, but unfortunately the changes required to measure those advertised speeds also make the tests much less realistic for the context of desktop workloads, so we'll continue to emphasize the more relevant low queue depth performance.

Samsung 980 Pro Cache Size Effects
Comments Locked

137 Comments

View All Comments

  • Tomatotech - Tuesday, September 22, 2020 - link

    NAND has a long way to go yet. It took a while for SATA SSDs to take off then they were bumping at the top of the SATA bandwidth for many years. Now the PCIe NVME drives are shooting up in data transfer speeds faster than the PCIe committee can bring out new speeds. NVME drives were all clustered at the top of PCIe 3.0 for a few years; PCIe 4.0 has only just come out and already much of the new headroom has gone. At this point the main bottlenecks are the drive controller chips and the PCIe standards themselves. 5.0 is being rushed out for a reason and it isn’t for GPU cards.
  • MFinn3333 - Wednesday, September 23, 2020 - link

    Modern SSD's endurance being small is mostly a myth. The last test I saw was on a Crucial BX500 120GB and it wrote 1.2PB on it before it gave out while a smaller 32GB hit around 250TB. The amount of voltage hitting the cells has dropped significantly over the years when they switched to Charged Trap Flash.

    The 600TBW is the minimum the drive will write.
  • Pinn - Tuesday, September 22, 2020 - link

    Curious about cooling. It looks bare but not near a GPU? Did you see thermal throttling?
  • Billy Tallis - Tuesday, September 22, 2020 - link

    The most recent SMART log for the 1TB 980 PRO indicates that it has spent about two minutes at or above the warning temperature (82C) but hasn't hit the critical temperature (85C). And that's out of about 14 power-on hours of testing. The SMART logs for the 250GB drive indicate that it has not hit its warning temperature.
  • back2future - Tuesday, September 22, 2020 - link

    What's the cells endurance and storage data stability compared at DWPD=0.3 on these high temperatures (179F, 82C) with long term comparison? What to expect on 5 year professional (high throughput) usage patterns?
  • Oxford Guy - Tuesday, September 22, 2020 - link

    “Samsung is abandoning the use of the two bit per cell (MLC) memory that has been the hallmark of the PRO product lines, and with the 980 PRO, Samsung is finally switching to three bit per cell (TLC) NAND flash memory.

    Along with switching to TLC NAND, Samsung has cut the write endurance ratings in half to 0.3 DWPD and dropped the usable capacities down to the typical TLC/EVO levels of 250/500/1000 GB instead of 256/512/1024 GB. TLC means the 980 PRO now relies on SLC caching for its peak write speeds, and write performance will drop substantially if the SLC cache is ever filled.”

    Finally!

    What consumers hope for: Worse endurance and worse performance. Higher profits for the company selling it.
  • Notmyusualid - Tuesday, October 6, 2020 - link

    @ Oxford - pretty much sums it up for me.
  • Whiteknight2020 - Tuesday, September 22, 2020 - link

    "Pro" means precisely and exactly, absolutely 0% of sod all. And always has. Either a product fits ones particular use case or it does not. I'm a "pro", in that I design, integrate an build Telco grade data centre systems deployments. I don't have a use for 600TBW to a 1TB drive, I'll never write that in it's useful life. I don't have a use for 6GB/s transfer rates. I do have a use for multiterabyte ssds at reasonable (1GB/s) transfer rates. For me, that would be a "pro" drive. For other workloads an entirely different "pro" profile is needed. There are few blanket use cases, except perhaps standard consumer laptop drives.
  • Oxford Guy - Tuesday, September 22, 2020 - link

    "'Pro' means precisely and exactly, absolutely 0% of sod all."

    You're correct and you're incorrect.

    You're correct in the sense that Samsung has rendered "Pro" no longer compelling by selling inferior TLC NAND in this product, undermining the established reputation of its Pro-labeled product line.

    You're incorrect in terms of this:

    "And always has."

    This is objectively flatly false. Samsung's Pro-labeled products have existed in the market for quite some time. They have an established reputation for targeting a specific level of performance and having other characteristics, like high-endurance MLC NAND.
  • Tomatotech - Wednesday, September 23, 2020 - link

    See the StoragePro review. This drive has some fantastic numbers under enterprise workloads.

    For home use? I’m not so sure about the value there. There’s a reason it’s called Pro.

    https://www.storagereview.com/review/samsung-980-p...

Log in

Don't have an account? Sign up now