When is a worthy alternative to Intel's Xeon finally going to appear? That is the burning question in the server world. If PowerPoint presentations from various ARM-based SoCs designers released earlier this decade were to be believed, Intel would now be fighting desperately to keep a foothold in the low end server market. But the ARM SoCs so far have always disappointed: the Opteron A1100 was too late, the X-Gene 1 performed poorly, consumed too much power, and Broadcomm's Vulcan project is most likely dead. This Ubuntu page is an excellent illustration of the current state of the ARM server market:

Discontinued products, many announced products which do not even appear on this page (we are in the middle of 2016, after all), and despite the fact that there is an ARM Server Base System Architecture (SBSA) specification, every vendor has its own installation procedure. It is still a somewhat chaotic scene.

Meanwhile, Intel listened to their "hyperscaler customers" (Facebook, Google...) and delivered the Xeon D. We reviewed Intel's Broadwell SoC and we had to conclude that this was one of the best products that Intel made in years. It is set a new performance per watt standard and integrated a lot of I/O. The market agreed: Facebook's new web farms were built upon this new platform, ARM servers SoCs were only successful in the (low end) storage server world. To make matter worse, Intel expanded the Xeon D line with even higher performing 12 and 16 core models.

But losing a battle does not mean you lose the war. Finally, we have a working and available ARM server SoC which has more ambition than beating the old Atom C2000 SoC. In fact, Cavium's ThunderX SoC has been shipping since last year, but you need more than silicon to get a fully working server. Firmware and kernel need to get along, and most libraries need to be compiled with platform-specific optimizations. So the quality assurance teams had a lot of work to do before Cavium could ship a server that could actually run some server software in a production environment. But that work has finally been finished. Cavium send us the Gigabyte R120-T30 running Ubuntu 14.04 server with a ThunderX ready Linux kernel (4.2.0) and ThunderX optimized tools (gcc 5.2.0 etc.).


Who is Cavium anyway? Even for those working in the enterprise IT, it is not a well known semiconductor company. Still, Cavium has proven itself as fabless network/security/storage and video SoC designing company. The company based in San José counts IBM, Juniper, Qualcomm, Netgear, Cisco among its customers.

With a net revenue of about $400 million, Cavium is about one-tenth the size of AMD. But then again, Cavium either reports small losses or smal profits despite heavy investments in the new ARMv8 project ThunderX. In other words, the company's financials look healthy. And Cavium did already design a 16-core MIPS64 Octeon Network Service Processor (NSP) back in 2006. So Cavium does have a lot of experience with high core count SoCs: the network processor Octeon III CN78xx has 48 of them.

Handling server applications is of course very different from network processing. A large amount of independent network packets creates a lot of parallelism, and more complex computation can be offloaded to co-processors. Still, Cavium is the first vendor that delivers an ARMv8 server chip with an impressive core count: no less than 48 cores can be found inside the ThunderX die.

To keep the design effort reasonable, Cavium based their first ARMv8 server processor, ThunderX, on the Octeon III CN78xx. We described this in more detail here, but the main trade-off is that Cavium used a relatively simple dual issue core. As a result, single threaded performance is expected to be relatively low. On the opposite side of the coin however, it is the first ARM SoC that has claimed throughput numbers in the realm of the best Xeon D and even midrange Xeon E5, instead of competing with the Atom C2000. It is the most ambitious ARMv8 SoC that has made it into production.

All of this gives us plenty of reasons to put the Cavium ThunderX through paces. And throwing in the latest Supermicro boards with the latest 12 and 16 core Xeon-Ds made it a lot more interesting ...

The ThunderX SoCs
Comments Locked


View All Comments

  • willis936 - Thursday, June 16, 2016 - link

    Are you sure that the there are more cores at lower clocks to keep voltage lower? Power consumption is proportional to v^2*f.
  • ddriver - Friday, June 17, 2016 - link

    Say what? Go back, read my previous post again, and if you are going to respond, make sure it is legible.
  • willis936 - Friday, June 17, 2016 - link

    Alright well if you don't understand why many slower cores are more power efficient even if there was a 0 cycle penalty on context switching then you aren't worth having this discussion with.
  • blaktron - Wednesday, June 15, 2016 - link

    48 cores of server processing on 16mb of l2 and 4 channels of RAM? What is this thing designed for. Will be like running single channel celerons as server processors, so decent hypervisor hosts are out, and so is any database work more complex than dynamic web pages.
  • Haravikk - Wednesday, June 15, 2016 - link

    Facebook is specifically mentioned as being interested in this, so dynamic web-pages is definitely a valid use-case here. HHVM for example is pretty light on memory usage (so is PHP7 now), especially in high demand cases where you're really only running a single set of scripts, probably cached in a compiled form, plus both scale really well across as many cores as you can throw at them.

    Things like nginx and MariaDB will be the same, so they're absolutely intended use-cases for this kind of chip, and I think it should be very good at it.
  • blaktron - Wednesday, June 15, 2016 - link

    With no L3 and slow RAM access I'm not sure where you think the scrips will cache. Assuming you ran them on bare metal (horrifying waste of compute) there would be enough, but if you had docker instances or quick spin vms doing your work (as 99% of web servers are) then each instance will only get the tiniest slice of cache to work with. It would be like running your servers, as I said, on a bank of celerons. Except celerons have L3 and don't carry 12 cores per memory channel.
  • spaceship9876 - Wednesday, June 15, 2016 - link

    Hopefully someone will release a server chip using 64 cortex A73 cpu cores, i'm pretty sure the cortex a73 will be more power efficient than xeon d. Xeon d beats cortex a57 in power efficiency but i'm pretty sure than cortex a72 will be similar and cortex a73 will beat it.
  • Flunk - Wednesday, June 15, 2016 - link

    ARM with ambition?

    I've heard that before, nothing came of it.
  • CajunArson - Wednesday, June 15, 2016 - link

    Interesting article. This does appear to be the first semi-credible part from an ARM server vendor.

    Having said that, the energy efficiency table at the end should put to rest any misconceived notions that ARM is somehow magically energy efficient while X86 isn't.

    Considering that Xeon E5-2690 v3 is a 4.5 year old Sandy Bridge part made on a 32 nm process and it still has better performance-per-watt than the best ARM server parts available in 2016, it's pretty obvious that Intel has done an excellent job with power efficiency.
  • kgardas - Wednesday, June 15, 2016 - link

    2 CajunArson: (1) you can't compare energy efficiency of CPUs made on different nodes. 28nm versus 14nm? This is apple to oranges. (2) Xeon E5-2690 *v3* is Haswell and not Sandy Bridge and it's not 4.5 years definitely.

Log in

Don't have an account? Sign up now