Earlier this week the crew over at Futuremark released a major update to their API Overhead testing tool, which is built into the larger 3DMark testing suite. The API Overhead tool, first rolled out in 2015, is a relatively straightforward test that throws increasingly large number of draw calls at a system to see how many calls a system can sustain. The primary purpose of the tool is to show off the vast improvement in draw call performance afforded by modern, low-level APIs that can efficiently spread their work over multiple threads, as opposed to classic APIs like DirectX 11 which are essentially single-threaded and have a high degree of overhead within that sole thread.

The latest iteration of the API Overhead test, now up to version 1.5, has added support for Vulkan, making it one of the first feature-level benchmarks to add support for the API. Khronos’s take on a low-level API – and a descendant of sorts of Mantle – Vulkan has been available now for just a bit over a year. However outside of a very successful outing with Doom, in the PC ream it has been flying somewhat under the radar, as few other games have (meaningfully) implemented support for the API thus far. By the end of 2017 we should be seeing some wider support for the API, but for the moment it’s still in the process of finding its footing among PC developers.

In any case, like OpenGL versus Direct3D 9/10/11 before it, there’s a lot of curiosity (and arguments) over which API is better. Now that Futuremark is supporting the API for their Overhead test, let’s take a quick look at how the two APIs compare here, and whether one API offers lower overhead than the other.

CPU: Intel Core i7-4960X @ 4.2GHz
Motherboard: ASRock Fatal1ty X79 Professional
Power Supply: Corsair AX1200i
Hard Disk: Samsung SSD 840 EVO (750GB)
Memory: G.Skill RipjawZ DDR3-1866 4 x 8GB (9-10-9-26)
Case: NZXT Phantom 630 Windowed Edition
Monitor: Asus PQ321
Video Cards: NVIDIA GeForce GTX 1080 Ti Founders Edition
NVIDIA GeForce GTX 1060 Founders Edition
AMD Radeon RX 480 8GB
Video Drivers: NVIDIA Release 378.92
AMD Radeon Software Crimson 17.3.3
OS: Windows 10 Pro

As a reminder for the API Overhead test, this is not a cross-system test or even a cross-GPU test. The purpose of the test is solely to measure overhead within a single setup. In practice it’s something of a combined GPU and driver test, as depending on where the bottleneck lies, the limiting factor can be CPU overhead from the driver or just outright hitting the limits of the GPU’s command processor.

The purpose of the test is to compare API performance on a single system. It should not be used to compare component performance across different systems. Specifically, this test should not be used to compare graphics cards, since the benefit of reducing API overhead is greatest in situations where the CPU is the limiting factor.

With that out of the way, let’s start somewhere in the middle of the pack with the GeForce GTX 1060 6GB.

3DMark API Overhead Feature Test: NVIDIA GeForce GTX 1060

Going into this I was not expecting Vulkan and DX12 overhead to be meaningfully different, so having run the GTX 1060 6GB first, it definitely caught me by surprise that Vulkan’s overhead was so much lower. The net result is that using the Vulkan API, the GTX 1060 can sustain 26.4M draw calls per second, 32% more than DirectX 12. And to be sure, this is consistent across multiple runs.

However before jumping to conclusions, let’s take a look at a couple of other cards.

3DMark API Overhead Feature Test: AMD Radeon RX 480

AMD’s fastest GCN4 card and card most comparable to the GTX 1060 is the Radeon RX 480 8GB. Running the API Overhead test on this card produces something notably different from the GTX 1060. Rather than finding Vulkan well in the lead, we get not quite the inverse: DX12 is holding a small lead at 26M draw calls versus 24.9M draw calls.

The two obvious differences here are the GPU and driver – in other words, the two things that matter the most – and is why cross-GPU results are not directly comparable. However it does go to show that whatever is causing Vulkan to perform better on the GTX 1060 is not a consistent factor. Switching things up can easily put Vulkan performance on the back foot.

3DMark API Overhead Feature Test: NVIDIA GeForce GTX 1080 Ti

Finally, to give the test as much GPU power as possible, I’ve also gone ahead and run it on NVIDIA’s recently released GTX 1080 Ti. This gives us a datapoint where GPU bottlenecking has been reduced as much as possible, and also gives us another datapoint with NVIDIA’s driver set.

The results still put Vulkan in the lead, but not by anywhere near what we saw on the GTX 1060. 32.4M calls versus 31.3M calls is a much narrower 4% difference between the APIs. What this does hint at is that on NVIDIA cards, the Vulkan API path has an edge in overhead, but even within just the NVIDIA ecosystem it’s not a massive difference. As with the AMD RX 480, for the GTX 1080 Ti this is essentially a draw between the two APIs. Which to be fair, is what we’d expect to find.

Overall the latest 3DMark API Overhead benchmark proved both reassuring, and more interesting than I was expecting to find. At a high level, neither Vulkan nor DirectX 12 hold a consistent lead with regards to API overhead, which indicates that both AMD and NVIDIA have done a good job optimizing their drivers and runtimes for this API. And more to the point, no matter the API used, it’s still vastly more efficient than DirectX 11, to the point where the draw call throughput is significantly greater than anything a developer could hope to use in the real world.

However the GTX 1060 results present an interesting anomaly, with the Vulkan API path showcasing noticeably lower overhead. I do have to stress that this is absolutely academic – these low-level feature tests are designed to test one small aspect of a GPU/system, and game performance won’t be anything like this – but it is an unexpected find that hints that Vulkan and DX12 may not be so neck-and-neck at all times on NVIDIA cards. The question we’re left to ponder is whether this is product of NVIDIA’s drivers, or if there’s something at the API level that just maps a bit better to NVIDIA’s command processor…

Comments Locked

47 Comments

View All Comments

  • Ian Cutress - Saturday, March 25, 2017 - link

    We hire based on expertise, not location.
  • geekman1024 - Saturday, March 25, 2017 - link

    Based on my many years' experience in wrangling with tech-support and customer service personnel -- you can tell the customer as many excuses as you like, but if in the end you don't deliver what the customer want, he/she'll just turn around and walk away - there are other places that can deliver.

    If your hiring policy lead to your inability in giving the readers what they want, it doesn't matter of how "expertised" you are.
  • Lord-Bryan - Sunday, March 26, 2017 - link

    Unfortunately, you don't pay for what you read here, so you can just go ahead and forget about this site, nobody will care.
    You can never get this much quality content elsewhere assuming you don't pay for analyst's like linley group, etc.
  • at80eighty - Sunday, March 26, 2017 - link

    The entitlement the users of this site has shown for the caliber of free content we get, is just mind shattering.

    You want your exacting standards met, start ponying up and earn the right to make them answerable to you
  • garbagedisposal - Sunday, March 26, 2017 - link

    Seriously trying to give some kind of life advice about tech support with a name like geekman1024? Are you joking?
  • Meteor2 - Sunday, March 26, 2017 - link

    Really? You care about _usernames_?
  • RiZad - Saturday, March 25, 2017 - link

    Not that its the same as what you are asking for but... here is my result with a 1700x and a GTX 1080.

    http://www.3dmark.com/aot/201803
  • unrulycow - Saturday, March 25, 2017 - link

    I wonder how DX12 and Vulkan compare to the DX12 derived language in the XBone and Mantle derived language in the PS4. I realize of course that it is basically impossible to test due to hardware differences.
  • Gigaplex - Sunday, March 26, 2017 - link

    I'd expect Vulkan to be fairly comparable to the PS4 since Vulkan itself is also derived from Mantle.
  • Manch - Monday, March 27, 2017 - link

    I wonder if the source of AMD's advantage with both DX12 & Vulkan is due to the PS4/XB1. When Mantle came out wasn't supposed to be a console like API for the PC or some jargon like that? GCN being the base on which both were developed seems to help them a good bit.

Log in

Don't have an account? Sign up now