Intel, AMD, VMware and…Aircrafts
"I need to setup a VI3 solution. Which CPU should I choose from a performance perspective? Intel or AMD?". I have heard this ad nauseam. If I was to get one single cent for every thread discussion I have seen on the net regarding the matter... I would be a billionaire. I have my opinion on the matter and I want to share it with you with this post.
Background
Intel and AMD are the two leading providers of x86 processors respectively with the Xeon brand and the Opteron brand (in the server space). This is not an introduction to what they do (Wikipedia might be your place if you are looking for that). At the time of this writing it must be noticed that Intel has introduced quad-core CPU support some 12 months ago more or less for the Xeon 5xxx family (previously Xeon DP for dual processor systems) and they have just announced quad-core CPU support for the Xeon 7xxx family (previously Xeon MP for multi processor systems). AMD has only been shipping dual-core CPU's so far and they have too just announced quad-core support across all their Opteron families (which includes 2P and 4P+ systems). So we will assume, throughout the discussion below, that they both provide quad-core CPU technologies.
Straight to the point, Xeons and Opterons have very radically different architecture implementations: Xeon has historically relied on the concept of the "Front Side Bus" where the sockets connect, through a bus, to the external memory controller. This picture should clarify the concept:
This architecture is referred to as a UMA architecture or Uniform Memory Access in the sense that each socket can get to any memory location in a uniform way (in terms of latency primarily). Consider that the latest chipsets now provide dedicated connections to each socket (as opposed to the shared connection that has been summarized in the picture), however it must be noticed that now 4 cores share the same socket thus increasing the level of contention.
On the other hand the Opteron processor has a very different architecture which is based on the concept of an integrated memory controller so that each socket (and memory controller) connects to the other sockets (and memory controllers) by means of a direct Hypertransport. Isn't it "cool"? Look at the picture below.
This architecture is referred to as a NUMA architecture or Non Uniform Memory Access meaning that, depending on where the data in memory is, a given CPU core could have a different latency to get to it. For example if the data that a CPU core on a given socket needs to access lives on a piece of memory directly connected to the integrated memory controller, access to it will be ideally very fast; if the data lives on a remote memory bank (i.e. one connected to the integrated memory controller of another socket) memory access will be slower.
You can imagine the memory access latency of a UMA system to be somewhat in the middle between the local memory access and the remote memory access of a NUMA system.
These are the facts. On top of this high level differences there are other important differences in the way Intel and AMD has implemented their solution. For example Intel has always been forced to use lately high capacity (dedicated and shared) caches among the cores to overcome the Front Side Bus contention inefficiencies. On the other hand the AMD architecture has not historically required high capacity caches due to their design to achieve comparable performance results.
How all this affects VMware ESX performance? Not too much in my opinion but more importantly not at the level for which you should bother. Let's see why.
Should I bother? I don't think so
My opinion, and my first rule of thumb, is... "number of cores being equal, if you spend more than 5 minutes worrying about these details you are basically wasting your time". This is not because raw performance is not important but simply because these guys are pushing what you can get out of the silicon to the limits (no matter what the architecture is) and it is very difficult to determine which one is faster over the other nowadays. There are market niches where AMD is still king of the hill (particularly those HPC memory intensive / floating point type of applications - sorry not an expert on the matter) but in general we are talking about commercial workloads running on top of a virtualization software layer which introduces a very unknown effect on performance.
Which leads me to my rule of thumb #2 that is... "if you meet/hear someone with a very specific idea of AMD being definitely better than Intel in a VMware environment or Intel being definitely better than AMD, disregard his/her opinions". Virtualization has radically changed the way we build our IT and you need to understand that all x86 server hardware has been studied and developed with the simple paradigm of 1server-1OS-1Application in mind. Every single piece of the various subsystems (cache levels etc etc) has been tuned to achieve the maximum performance, over the last few years, for that legacy context. When you scramble all that with the new paradigm that is 1server-1hypervisor-nOSs-nApplications you are basically using a hardware platform (including the CPU) that has been tuned for the last decade to do something else. There are a very small set of people in this industry (I would say perhaps 30 or 40 and they typically work for CPU vendors, systems vendors and virtualization software vendors) that are working day and night to understand the effects that virtualization is having on the current generation of hw systems. And quite frankly the most extreme comment I have heard from them was a "for this particular workload running in a virtual machine, we have noticed that
Without getting into the problem of religious wars which I think are pretty ridiculous, the problem with people claiming very definitive statements of one being better than the other is that they are convinced of what they are saying. Most likely they have both technologies running in-house and based on their limited experience (not limited time wise - but limited projects-wise) they claim that "based on what they have seen" A is better than B. The problem with this is that it is ENORMOUSLY difficult to run a benchmark that allows an apple to apple comparison between two different systems to determine which one is the best. There are dozens (perhaps hundreds) of variables in such field experiences / benchmarks that are just not valid scientific method to valuate such a complex matter. And I don't even want to talk about people that benchmark their 4-socket enterprise servers running a file copy or using SiSoft Sandra (which would be like benchmarking the speed limit of a Ferrari with the test taking place in Milan downtown at 8am ... what would you expect? A Fiat could turn out to be faster than a Ferrari given this variable). I want to talk about subdle variables that might lead you to think something that is not real in facts. A good (real) example would be a customer of mine complaining about a slow application running within a vm. He was complaining about the fact that the batch job running inside a vm would take twice the time to complete compared to the same job running on a similarly spec'ed physical system. It turned out that running the job at different points in time during the day provided (very) different results. This was nailed down to be an environmental problem (such as SAN and network utilization at different hours) but let me tell you: 99% of the people testing an AMD based system at 8am in the morning with this batch (elapsed time: 2 hours) and an Intel based system (same config, same everything....) at 11am (elapsed time: 3 hours) would definitely point their browser onto the VMware forum with a nice post that says "I know for sure that an AMD system is 50% faster than an Intel based system". Missing the point.
However I am not saying that both AMD and Intel are equal performance-wise. What I am saying is that:
- We know so little about the effects that virtualization is posing on the hardware (we are just scratching the surface now of these implications)
- Both Intel and AMD are pushing the silicon to the extreme of laws of physics
- Both Intel and AMD are leapfrogging each other every 6 months
... that to me it is a waste of time discussing which one is better than the other. What you need to keep in mind is that they are damn good pieces of technology and you can't go wrong with either one or the other.
This leads me to the "IT artists" which is the extreme, in my opinion, of these "home-made experts". The IT artists are those that argue AMD being better than Intel because of its "native quad-core design". As an IT guy I am interested in three things CPU-wise:
- Absolute performance
- Price
- Power consumption
Quite frankly how AMD or Intel is giving me the above... is a technical detail I am not interested in. The IT artists however tend to articulate that AMD has no FSB and the way Intel can keep up in terms of performance is using big caches. This is very true but the real point is: as long as Intel is giving me that big cache at a fraction of the costs AMD would need to charge me for... should I care? I don't work for the fashion business so I am not interested in how "elegant" the AMD design is Vs the "cumbersome" Intel design. What I want is highest performance, at the lowest price, at the lowest power consumption. Period. I am not bashing AMD here to promote Intel. I think AMD is doing wonderful things to make this x86 platform progress as I tried to point out here but the problem is that it seems that "fashion" is easier to sell and market than the "real" stuff.
And before you get me wrong I am not saying that an integrated memory controller is not a smart choice (Intel will get there anyway) and certainly those 30 - 40 people above could entertain us some 2 days talking about the performance implications of having a FSB Vs an integrated memory controller. But they are not IT artists, they start from the layout to articulate from a scientific perspective why and how it affects performance. IT artists just look at the layout and say "the layout is cool so it must be better".
Rule of thumb #3 is "when you meet an IT artist, don't pay too much attention to what she/he says".
Conclusions
I like to close with an analogy here. Think at the next purchase of hardware as a journey from Milan to New York. Where the vendor you are buying from being the airline, the aircraft you are flying with being the system/server and the engines of the aircraft being the CPU's.
So my question is... when you need to buy a ticket... do you start from which engine provides the best performance to search then for aircrafts that use that engine to determine which airlines use those aircrafts... and you buy a ticket from them? Well... no offense but if you do, I think you need a doctor.
When I buy a ticket I care more about the airline that provides me the best service (at the right price obviously). I want to buy a ticket from an airline that appreciates my business, that is there to support me if I need it, that is able to make my life easier and my journey better if I go with them. This includes having chosen the right aircraft for that journey for my comfort. There might be instances however where, even a good airline, is using obsolete and odd aircrafts and, although they are alright and supportive as an organization, you might want to fly with another company just because they have better aircrafts. What I am pretty sure of though is that if the airline suites your need and they are using good and comfortable aircrafts that meet your standards... there is no way you don't want to fly with those just because you think they are not using the "coolest" engines. After all, assuming you can get to destination 20 minutes in advance with the cool engines, your journey might be a nightmare anyway (especially if your take-off gets delayed by 4 hours ... what are those 20 minutes going to buy you in the end?)
What I am trying to say here is that when you decide to buy a new server from a system vendor you should first look at the commitment that this system vendor has with regards to your business (that is virtualizing your system). A good system design is of course important and needs to meet your standards and requirements. However good management tools, an understanding of what your objective is and the capability to complement a hardware sale with the know-how to implement what you need are other important characteristics along with a vision on where this industry is going so that the vendor can help you take the right decisions. A vendor or partner that you trust basically.
Such systems vendors have done their diligent homework to understand what CPU to use in any given system and you should trust their choice (like you trust Airbus or Boeing for having chosen a specific engine for a specific aircraft). After all you need to remember that what a system vendor wants to do is to sell you something that performs well at the lowest cost and lowest power consumption. In the final analysis they want happy customers otherwise customers will not come back.
Don't waste your time on the engines: it's a detail of a much bigger picture (and the detail right now is not even important given that the two options are excellent). Look at the forest not at the tree. When booking a journey, ask yourself which frequent flyer card you should apply to ... not which engine is faster.
Massimo.