back to article China's corruption crackdown killing off Unix

China's crackdown on Bo Xilai and other scandal-hit Communist Party outcasts may be making Unix systems less attractive behind the Great Firewall. That's the opinion of EMC's president for Greater China, Denis Yip, who last week told attendees at an EMC Forum event in Hong Kong that the “political environment” across the …

COMMENTS

This topic is closed for new posts.

Page:

  1. Charles Manning

    It always starts at the low end.

    Japan first made crap toys that broke the day after Christmas, rough cheap cars and cheap tinny transistor radios. Now, quality, quality, quality.

    Same for Korea who have moved from crappy brands to top end stuff.

    No doubt the same will happen in China, but just faster. Huawei started making candybar phones and less than spectacular cell/networking equipment. Now they're producing reasonable smart phones and have gone up a few notches in infrastructure gear. Within a year or so we'll start seeing top shelf product.

    1. earplugs

      Re: It always starts at the low end.

      What year is this? XiaoMi is already worth almost twice Nokia.

      1. Robin Bradshaw

        Re: It always starts at the low end.

        The fluff in my bellybutton is worth more than Nokia.

        1. Dan 55 Silver badge
          Trollface

          Re: It always starts at the low end.

          "The fluff in my bellybutton is worth more than Nokia."

          More attractive user interface as well.

        2. Anonymous Coward
          Anonymous Coward

          Re: It always starts at the low end.

          Do you bath in a gold mine then?

          My Nokia shares are worth nearly double what I paid for them a year ago...

      2. Daniel B.
        Meh

        Re: It always starts at the low end.

        "What year is this? XiaoMi is already worth almost twice Nokia."

        Had you said this in the pre-Elopocalypse world, I would be shocked. Right now, I'm just meh.

    2. JohnG

      Re: It always starts at the low end.

      "Huawei started making candybar phones and less than spectacular cell/networking equipment."

      Huawei's origins were somewhat grubbier than that - they started out making counterfeit Cisco kit.

      1. Alan Brown Silver badge
        Mushroom

        Re: It always starts at the low end.

        "Huawei's origins were somewhat grubbier than that - they started out making counterfeit Cisco kit."

        Not quite.

        They were Cisco's contract manufacturer for many years, then started selling the same kit under their own brand (naughty). It's no different to a number of other "knockoffs" which are off the same production line but not branded.

        This was the reason Cisco moved from open updates of their software to a subscription licensing model (Sun did the same thing for the same reasons)

        In any case, Huawei no longer manufacture for Cisco and their current kit shares no common hardware internals with Cisco (Not suprising given that as a rule Cisco gear is grossly underpowered for the price).

        (I've just spent 3 months evaluating switch/router kit and just about everyone out there is selling kit that's twice as powerful as Cisco for half the price, even before discounts are taken into account, and the differences stay much the same after discounts are factored in (noone EVER pays list price unless they're terminally stupid). Brocade in particular is a very attracitve proposition for enterprise kit if you don't want to buy Huawei because of politics)

    3. Anonymous Coward
      Anonymous Coward

      Re: It always starts at the low end.

      China are just following the rest of the world, but a bit behind the curve as usual.

      Most enterprises elsewhere are already a long way down the road of replacing legacy UNIX boat anchors with more cost effective Wintel solutions...

  2. frank ly

    Switching from big iron to x86 virtualisation

    What does that have to do with Unix??

    1. A Non e-mouse Silver badge

      Re: Switching from big iron to x86 virtualisation

      I suppose the old adage of "No one got fired for buying IBM" has a rather macabre connotation in China now...

    2. Peter Gathercole Silver badge

      Re: Switching from big iron to x86 virtualisation

      Indeed, the sentence that reads "people are not willing to spend big money on mainframes – the Unix big guns" indicates that Dennis Yip has only a limited grasp of what he is talking about.

      From a cursory examination, I would suggest that in his world there is just Wintel, and a single category of "everything else".

    3. Kebabbert

      Re: Switching from big iron to x86 virtualisation

      Linux is fine for low end server work, or for large clusters doing parallel HPC computations such as the SGI Altix server. x86 servers are also getting more and more powerful, so Linux can take over low end work loads. But there are no high end SMP Linux servers for sale and has never been, so for high end you must go to Unix or Mainframes. You have no choice. Enterprise work loads require large SMP servers with as many as 16 or 32 cpus, and no one has ever sold such large Linux servers. But for cluster workloads, HPC Linux servers are fine - but no Enterprise use HPC servers, only SMP servers.

      1. Anonymous Coward
        Anonymous Coward

        Re: Switching from big iron to x86 virtualisation

        This simply not true, and your SGI and IBM reps will be happy to inform you otherwise.

      2. Roo

        Re: Switching from big iron to x86 virtualisation

        "But there are no high end SMP Linux servers for sale and has never been, so for high end you must go to Unix or Mainframes."

        The article said "UNIX", Linux is not UNIX.

        "You have no choice."

        That is bollocks. There is more choice now than there ever has been, because slotting 8+ cores onto a die is routine these days.

        "Enterprise work loads require large SMP servers with as many as 16 or 32 cpus, and no one has ever sold such large Linux servers."

        That is also bollocks, here are just two reasons why.

        1) "Enterprise" workloads come in all shapes and sizes. Frequently it is memory capacity that is the problem, not CPU count.

        2) I have been working on Linux boxes with >= 16 cores on and off for the last 8 years or so. The manufacturers of these boxes include the usual suspects (Dell, HP, IBM).

      3. fch

        Re: Switching from big iron to x86 virtualisation

        Refresh your tech knowledge. There's quite a few options in the x86 space these days that offer 16 or 32 CPU cores. Even if you count chips / sockets, you have a range of choice in the 8-socket space (remind me there, how many CPU sockets did a T5-8 have, again ... ?).

        Yes, there's not that many x86 servers out there that can have 32 CPU sockets. If that's what counts for you, go IBM / Fujitsu / Oracle.

        No, there's not exactly a huge number of usecases for iron of that size. A bunch of two-socket boxes behind a load balancer does, these days, often beat the "big fat box" approach not just on latency/throughput but much more so on price, and/or price/performance.

        Or, to put it differently, CPU-performance-wise, what a Sun E10k did in 1997, a Samsung Galaxy 4 smartphone chip does today. Probably more. You just no longer need 64 CPUs for this.

        Enterprise workloads have grown nowhere near as much as have the abilities of "cheap" (x86) hardware.

        That's the bane of the non-x86 vendors; same number of sharks in an ever-shrinking pond. Are those "big" boxes faster / can they process more data in shorter-a-time ? All yes, but the deciding question really is: "what box do I need for my workload ?".

        1. Kebabbert

          Re: Switching from big iron to x86 virtualisation

          "....Refresh your tech knowledge. There's quite a few options in the x86 space these days that offer 16 or 32 CPU cores. Even if you count chips / sockets, you have a range of choice in the 8-socket space (remind me there, how many CPU sockets did a T5-8 have, again ... ?).

          Yes, there's not that many x86 servers out there that can have 32 CPU sockets. If that's what counts for you, go IBM / Fujitsu / Oracle...."

          I am glad that you agree with me: There are no 16 or 32 socket Linux SMP servers for sale, and has never been. The T5-8 has 8 sockets. The Oracle M6 has 96 sockets, the Fujitsu M10-4S has 64 sockets. The IBM P795 has 32 sockets. HP has a Itanium Superdome/Integrity server with 64 sockets. There have never been a 32 socket Linux SMP server for sale. If someone objects, I invite him to post links to a 32 socket Linux SMP server. Good luck, there has never existed such as server. Sure, the SGI Altix 2048 core server is a HPC cluster, but there are no 32 socket Linux SMP servers for sale. The ScaleMP 2048 core Linux server is running a single image kernel on a cluster. The latency is so bad it is only fit for HPC workloads such as number crunching, just as the SGI Altix cluster.

          The >32 socket server market is very very lucrative and high margin. The x86 business is low margin, and Larry Ellison has declared that he does not care if the x86 business at Oracle dies, because the margin is so low. IBM and Oracle does high margin business, that is where the big bucks are. For instance, the old 32 socket IBM P595 used for the old TPC-C world record costed $35 million list price. $35 million. I kid you not. That is some serious money. The largest IBM Mainframe has 24 sockets, and it is also very lucrative and one of IBM's big cash cows.

          If you Linux supporters say that I am wrong, please show us a link to a 32 socket Linux server. No one has ever showed me such links. Never ever. They claim I am wrong, but... no links. Lot of talking, but no proof. I am wrong, then it is easy to make me shut up: show us a link to single 32 socket Linux server. :)

          1. Anonymous Coward
            Anonymous Coward

            Re: Switching from big iron to x86 virtualisation

            Here we go:

            http://www.sgi.com/products/servers/uv/models.html

            256 socket NUMA single system image coherent global shared memory. It is not a distributed memory HPC cluster.

            IBM:

            http://www-03.ibm.com/systems/power/hardware/795/specs.html

            The 795 has 32 sockets and SuSE Enterprise Linux is one of the supported systems.

            1. Kebabbert

              Re: Switching from big iron to x86 virtualisation

              @Roo

              "...If your definition of x86 servers can stretch (a lot) the Cray XC-30 might be of interest. It ships with the Cray Linux Environment, a cabinet can hold 384 sockets (3072 Xeon E5 cores), infinband I/O. You can add a lot of cabinets too. At the lower end you have the SeaMicro boxes ranging from 64 to 256 sockets (and I have seen them referred to as 'servers')...."

              The Cray XC-30 is a cluster. It is a HPC cluster used for number crunching. Have you seen the workloads the Cray tackles? All embarassingly parallel workloads, running on lot of nodes, on a cluster. Cray does not make SMP server (a single big fat server, running for instance databases). Cray makes computing clusters, not Enterprise servers running Enterprise work loads. You will never see such HPC clusters running big fat Oracle databases, for instance.

              .

              .

              Anonymous Coward,

              "....256 socket NUMA single system image coherent global shared memory. It is not a distributed memory HPC cluster.... The IBM P795 has 32 sockets and SuSE Enterprise Linux is one of the supported systems...."

              First, ccNUMA servers are a cluster. Not SMP servers, nor close to SMP servers. They can not handle SMP workloads, they are a cluster:

              http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access#NUMA_vs._cluster_computing

              Second, the IBM P795 is not a Linux server. It is an AIX server that someone has compiled Linux for. I doubt anyone runs Linux on it, because the P795 is so expensive. It is better to run Linux on cheap x86 servers. Besides, Linux would never be able to scale to 32 sockets. HP had their "Big Tux" Linux server, which was the 64 socket Itanium Integrity (or was it Superdome?) server that they compiled Linux to, and Linux had something like ~40% cpu utilization using 64 sockets. Linux scaled so bad on 64 sockets, that when HP sold Big Tux, HP only allowed Linux to run in a partitioned server. The biggest supported Linux partition on Big Tux was 16 cpus. If Linux scaled well, HP would have supported 64 socket partitions too. But they didnt. 16 cpu Linux partitions did not work that well, either. If you look at modern benchmarks of Linux on a 8-socket x86 server, the cpu utilization is quite bad. For instance, SAP benchmarks show 87% cpu utilization on a 8-socket server. 16 socket would give... 60% cpu utilization, I guess. And 64 sockets does give ~40% cpu utilization in confirmed benchmarks. I am convinced the IBM P795 Linux offering is very limited in terms of Linux scalability, like, it has 40% cpu utilization, or P795 is only allowing max partitions of 8-16 cpus.

              So, no, there are no Linux SMP alike servers. Some people have compiled Linux to big Unix servers, but that does not make them Linux servers. For instance, you can run Linux on a IBM Mainframe with 24 sockets, but that does not make the IBM Mainframe, a Linux server.

              .

              If you look at the RAM latency of a true SMP server, it has uniform latency from every cpu. No matter which cpu you are using, it accesses RAM as fast as every other cpu.

              SMP alike servers, for instance the Oracle M9000 SPARC server with 64 sockets, has a worst case latency of 500ns, which is quite bad. But best case is something like 100ns or so. So the spread is tight, no big difference between worst case and best case. The Oracle M9000 is not a true SMP server, because there is some difference in latency. But that does not matter, see below.

              If you look at the latency of a NUMA cluster like the SGI Altix 256 socket Linux server, the worst case latency is something like 10.000ns, or was it 70.000ns? I can't remember, but I know it was above 10.000ns, which is catastrophic. This changes everything. If you develop for the SGI server, you must allocate data close to the current node, and you design your software differently than for a SMP server - otherwise the performance will be extremely bad. In effect, you design your software exactly as if it was a cluster: you allocate data close to the current node, etc. And if you look at the workloads the customers are buying SGI for, it is for HPC workloads, and other clustered workloads. If you do SETI number crunching, each node does not have to talk to other nodes, that is a typical parallel workload that NUMA clusters handles fine.

              If you study the new Oracle M6 server with 96 sockets, and ~10.000 threads, it is a mix of 8-socket SMP servers, connected with NUMA connections. But, the worst case latency is again, only 2 hops or so, just like in the Oracle M9000 server. If you need to access data, it will not take you more than 2 hops to reach it, which is fast. In effect, when developing for the M6 server, you design your program as it was a true SMP server. You dont need to allocate data to close nodes, etc, just develop your software as normal. So, Oracle M9000 and Oracle M6 runs SMP workloads just fine, and you dont have to redesign your software. Just take your current Solaris binary, and copy it to the M6 server and it will run fine. Try to do that on the Linux SGI cluster, and it will show extremely bad performance unless you redesign your program. Oracle M6 will run SMP workloads such as the Oracle Database in very large configurations. Databases is all Oracle cares about. So, in effect, the Oracle M6 and M9000 behaves as if they were true SMP servers, and you dont need to redesign your software to run on them. This is not the case using the Linux SGI clusters, they can never show good performance running large database configurations, with worst case latency of 70.000ns.

              So, no, there are no Linux 32 socket servers for sale. Sure, IBM has one AIX P795 server they offer Linux on, but that does not make it a Linux server. And IBM offers Mainframe with 24 sockets to run Linux, but that does not make the Mainframe a Linux server either. If someone can show a link to Linux 32 socket server, I would be very surprised, because no one has ever manufactured such a Linux server. And NUMA servers dont count, they are just a cluster. Anyone can make a cluster, basically, just slap on lot of PCs on a fast switch and you are done.

              So, I invite anyone to show a Linux server with 32 sockets. There have never been until today. Why? Linux scales very bad, shows benchmarks from HP on their HP-UX 64-socket server. Linux can not go beyond 8-socket servers today. And Linux does not even handle 8-socket servers well, just look at such benchmarks, and read about "cpu utilization".

              1. Roo

                Re: Switching from big iron to x86 virtualisation @ Kebebbert

                "First, ccNUMA servers are a cluster. Not SMP servers, nor close to SMP servers. They can not handle SMP workloads, they are a cluster:"

                1) A ccNUMA system can provide a shared address space to multiple processors.

                2) A cluster typically consists of a bunch of machines that do NOT have a shared address space.

                3) Even Xeon cores sharing a socket can see latency vary from ~100ns to over 300ns, gee look at that, NUMA.

                4) the hardware that you assert as being "SMP" capable is in fact NUMA hardware and therefore a "Cluster" by your own definition.

                5) Clusters are capable of providing ccNUMA, the degree of success depends on the tech used of course.

                "If you look at the RAM latency of a true SMP server, it has uniform latency from every cpu. No matter which cpu you are using, it accesses RAM as fast as every other cpu."

                Even with "true SMP systems" (as you define them), latency is not uniform because of cache-misses, locks, cache-cohrence protocols etc. I remember the feeling of disappointment quite clearly when I found out the hard way over 20 years ago.

                "So, I invite anyone to show a Linux server with 32 sockets".

                You have already been given plenty of examples, you chose to assert they are not Linux servers which is pretty dumb seeing as they are servers that run Linux.

                What you appear to be asking for is a Linux server with 32 sockets that has a worst/best case memory latency ratio <= 3. Just to give you a hint, folks have measured 100-300+ ns memory latency on a pair of Xeon E5 cores sharing a socket.

                I'll put it this way : Address spaces are about as flat as the Earth. If you write your code assuming a flat address space you should expect it to perform badly because even on single core machines we have caches to exploit these days.

                1. Kebabbert

                  Re: Switching from big iron to x86 virtualisation @ Kebebbert

                  @Roo

                  Of course you should consider caches and align vector properly, when programming SMP servers. Of course there will be differences in best worst case latency in SMP, because of when cache pipeline stalls, etc. Who on earth would believe the opposite???

                  My point is that when you program for a NUMA cluster with worst case latency of 10.000ns or more, you need to carefully redesign your software. You can not copy your binaries to a NUMA cluster and expect it to perform well, it won't.

                  But for a SMP alike server such as Oracle/Sun servers, you can copy your binary to it and expect it to perform well. That is why Oracle is building the M6 server with 96 sockets and ~10.000 threads and 96TB RAM. Oracle intends to run Databases on it, as Larry Ellison said: he does not believe in SAP Hana RAM databases, because Oracle databases will run as good as (if not better because of more RAM) than the Hana RAM database. You will never see anyone run a database on a NUMA cluster, that would drag down the performance to a halt.

                  .

                  "...You have already been given plenty of examples [of 32 socket Linux servers], you chose to assert they are not Linux servers which is pretty dumb seeing as they are servers that run Linux...."

                  I have been given two different Linux NUMA servers. Ever. And I myself have given example of a third NUMA cluster, that is the ScaleMP server. As we all know, NUMA servers are clusters. And they dont run SMP workloads, such as huge database configurations. They are only doing number crunching.

                  I have also been given ONE single example of a 32 socket server, that is the IBM AIX Unix P795 server. I myself has given an example of a 64 socket server, the HP-UX Itanium Superdome server. But as we all know, these Unix servers, are.. Unix servers. Linux scales awfully bad on the HP-UX server, it is hardly supported (only up to 16 cpus). The IBM P795, I expect Linux to scale as bad on it too. I would be surprised if any customer in the world runs Linux on such an expensive server. For the price of one single POWER7 cpu, you could buy a cheap 4-socket x86 server. Nobody runs Linux on an expensive IBM Unix AIX P795 server, I am convinced. The predecessor, the old 32 socket IBM P595 server for the old TPC-C record costed $35 million list price. Who would run Linux on a $35 million server? Why not buy a bunch of cheap x86 servers?

                  So again: I invite anyone to show links to a Linux 32 socket server, which is not an existing Unix server. We are talking about enterprise, and enterprise runs big databases, not number crunching. Is there any Linux 32 socket server out there? No? And it has never been. No matter how harsh language you use, it wont change the fact. Put up, or shut up. Show us the links, the proof.

                  1. Roo

                    Re: Switching from big iron to x86 virtualisation @ Kebebbert

                    "But for a SMP alike server such as Oracle/Sun servers, you can copy your binary to it and expect it to perform well"

                    That is only a given if your binary is solving a trivially parallelized problem, and in those cases a cluster will do just as well. I admire your loyalty to Oracle's hardware, at the end of the day the majority of high volume low latency number/data crunching appears to be done by heavyweight Xeon boxes abetted by FPGAs & GPUs at the moment. By your rather weird classification system those 4-32 core boxes would be classed as clusters

                    because they are NUMA systems with > 300ns worst case latency.

                  2. Roo

                    Re: Switching from big iron to x86 virtualisation @ Kebebbert

                    "As we all know"

                    That's only true in a universe with a population of one, ie: yourself.

                    "NUMA servers are clusters."

                    "NUMA" and "Clusters" are orthogonal concepts. NUMA refers to the properties of memory access, clusters consist of a set of tightly coupled but otherwise self-contained systems that co-operate to act a single system. The original VAX Clusters did this to enhance availability and throughput.

                    Some NUMA systems can run as clusters, and vice versa.

                    Traditionally a cluster is a bunch of hardware hooked up via some interconnect where the nodes do not share memory.

                    "I invite anyone to show links to a Linux 32 socket server, which is not an existing Unix server"

                    Ah, so a 32 socket server that is running Linux doesn't count because it can run Windows, HP-UX or AIX too ? You must be a fully paid up member of the Flat Earth Society.

                    1. Kebabbert

                      Re: Switching from big iron to x86 virtualisation @ Kebebbert

                      "....[You can copy your Solaris binary to a SMP server and expect it to perform well without rewriting it] That is only a given if your binary is solving a trivially parallelized problem, and in those cases a cluster will do just as well...."

                      Not at all. I dont know how much you know about computer science or programming, but these Oracle SMP alike servers are not only running NC-complete problems. If you believe that, I suggest you study the customers (enterprise companies), and what they use the Oracle servers for: typical SMP workloads such as big databases.

                      And then I suggest you compare the customers for HPC clusters such as the SGI Altix server and their workloads. You will find the customers are researchers, weather forecasting, oil companies etc, and they all do number crunching. No one runs big databases or other SMP work loads.

                      I dont know how many times I need to say this? Just compare the workloads and see what the SGI Altix server is used for, and see what Oracle and IBM and HP unix servers are used for. You will there is a big difference. Do you understand now, at last? These servers are used for different tasks, one for SMP things, and the other for HPC number crunching.

                      .

                      "....Ah, so a 32 socket server that is running Linux doesn't count because it can run Windows, HP-UX or AIX too ? You must be a fully paid up member of the Flat Earth Society...."

                      I am trying to say that these big Unix servers made by IBM, HP and Oracle, are Unix servers. Benchmarks from HP shows that Linux does not run well on 64-socket SMP servers. My point is that there are no Linux vendor designing Linux 32 socket SMP servers. No one. If you need 32 sockets, you need to go and buy an large expensive Unix server and compile and install Linux for it - and pray Linux does not fall apart on the server. I would hardly classify this configuration as a "Linux server". These Unix servers were built for Unix, and Unix scales well on these servers. Linux is hardly supported on them, on the HP-UX server, Linux is only supported up to 16 cpus - do you really call it a "Linux server"???

                      Just because my car can run on race tracks, I dont belive it is a Formula1 car - do I? Exactly what is it that you have difficulties in understanding? You dont agree that all these large Oracle/IBM/HP 32-64 socket servers were built for running Unix and that HP has provable big troubles running Linux above 16 cpus? What is so difficult to understand?

                      Do you still believe Linux scales well on 32 socket servers? If you do, on which server have you seen those benchmarks? Not on HP's servers, because they have documented bad scaling. Maybe you have seen Linux benchmarks on the IBM P795 server? Can you show us them benchmarks? If you can not show us benchmarks on IBM P795, which server have you seen good Linux benchmarks on? Not HP. Not IBM. Not Oracle. Hmmm... there are no other 32 socket vendors. I wonder how this "Roo" guy can claim Linux scales well on 32 sockets - because it doesnt. Maybe he is talking right out of his nose and making up things, and have a vivid imagination?

                      1. Roo

                        Re: Switching from big iron to x86 virtualisation @ Kebebbert

                        "Just compare the workloads and see what the SGI Altix server is used for, and see what Oracle and IBM and HP unix servers are used for. You will there is a big difference. Do you understand now, at last? These servers are used for different tasks, one for SMP things, and the other for HPC number crunching."

                        Given the >3x performance advantage of the Xeon cores over the M9000 cores in SPEC rate figures I think that very few people would choose the M9000 for the kind of compute intensive workloads because it would need 3x as many sockets to achieve the same result with perfect linear scaling (ie: not gonna happen).

                        I can't help but notice that these benchmarks, that you claim show Linux does not scale well, are running on HP hardware, meanwhile all the examples of great scalability seem to come from the SPARC/Solaris hardware. Perhaps the HP hardware was the root cause of the poor scaling in those benchmarks you mention. The only way you can prove that point is to run Linux & Solaris & HP-UX on the same hw platform, which so far you have not done.

                        Let me know if you can find some benchmarks that test scalability for Solaris & Linux running on identical SPARC 16/32/64 socket hardware.

                        1. Kebabbert

                          Re: Switching from big iron to x86 virtualisation @ Kebebbert

                          @Roo,

                          "...Given the >3x performance advantage of the Xeon cores over the M9000 cores in SPEC rate figures I think that very few people would choose the M9000 for the kind of compute intensive workloads because it would need 3x as many sockets to achieve the same result with perfect linear scaling (ie: not gonna happen)...."

                          Wrong conclusion. If the Xeon core is 3x faster, then it does not follow that M9000 needs 3x more sockets. Because, there is a huge difference between core and cpu. Maybe the M9000 cpus has very many cores, maybe 1000s of them. Then you dont need 3x more _sockets_. Or, if the M9000 cpu has only one core, and the Xeon has 1000 cores, then M9000 need many more than 3x sockets to catch up on one Xeon.

                          But it is true that Xeon is faster at number crunching. The reason is becuase SPARC is designed for Enterprise. Which means SPARC has better RAS (if a cpu instruction was errorneous the cpu can rollback and replay the instruction just like in IBM Mainframes, Xeon can not do this), and the SPARC is better at SMP workloads because it scales better. You can run large databases on M9000, but not on Xeon servers.

                          .

                          "....I can't help but notice that these benchmarks, that you claim show Linux does not scale well, are running on HP hardware, meanwhile all the examples of great scalability seem to come from the SPARC/Solaris hardware..."

                          HP has good scalability on the same 64 socket hardware, when running their own Unix called HP-UX. They offer 64 cpu configurations when running HP-UX on the Big Tux server. But when running Linux on the same Big Tux server, the largest supported Linux cpu configuration is 16 cpus. No 64 cpu Linux configurations are offered. Why is that? Is it because Linux has problems utilizing 16 cpus? With larger configurations there will be too much support problems? So, where are those superior Linux 32 socket benchmarks?

                          .

                          "....Let me know if you can find some benchmarks that test scalability for Solaris & Linux running on identical SPARC 16/32/64 socket hardware...."

                          I know that Linux runs on small 1-2 socket SPARC workstations, but it would be far fetched to believe you can move the same Linux to 32 socket SPARC servers without problems. That would need lot of tailoring and recompiling, and redesigning of the Linux kernel. So dont expect to see Linux running well on larger SPARC servers. SPARC servers have traditionally been targeted to scalability, not number crunching. Many SPARC benchmarks are about scalability, it is the strength.

                          However, there are Solaris and Linux benchmarks running on nearly identical x86 hardware. When using few sockets, 1-2 sockets, Linux wins sometimes and Solaris sometimes. But when you go up to 8 sockets (which is very small servers) Solaris wins easily. The more sockets, the better Solaris scales. On 1-2 socket servers Linux wins most of the time (I suspect). On small handheld devices, Linux wins I suspect. But on larger server, 8 sockets and upwards, Solaris wins - because that is Solaris domain. Solaris has long been made for scalability on large servers, and if Solaris did not win on large servers, that would be an indication something is wrong.

                          1. Roo

                            Re: Switching from big iron to x86 virtualisation @ Kebebbert

                            "Wrong conclusion. If the Xeon core is 3x faster, then it does not follow that M9000 needs 3x more sockets. Because, there is a huge difference between core and cpu. Maybe the M9000 cpus has very many cores, maybe 1000s of them. Then you dont need 3x more _sockets_. Or, if the M9000 cpu has only one core, and the Xeon has 1000 cores, then M9000 need many more than 3x sockets to catch up on one Xeon."

                            Well, I could have worded it better and included the core counts, but I don't think a comparison of core counts really makes much sense - given that you were arguing about socket counts. The fact remains that a 64 socket UV1000 running a stock x86_64 release of SUSE Linux is 3x faster in SPEC rate (int and fp) than a 64 socket M9000 running an specialised cut of Solaris.

                            Another point to consider with respect to your assertion that you will get good scaling from unmodified binaries on an M9000 is that it relies on running a huge number of threads to achieve high system throughput at the expense of a big hit in single-thread throughput. Legacy binaries are usually tuned to run on high-single-thread throughput systems, so I would expect to see better scaling and throughput from a system with a high-throughput cores (eg: Xeon) vs low-throughput cores (SPARC Tx).

                            With the SAP benchmarks - were they running the apps in different partitions/containers/virtual machines or were all the processes sharing the same OS instance ?

              2. This post has been deleted by its author

          2. Anonymous Coward
            Anonymous Coward

            Re: Switching from big iron to x86 virtualisation

            "If you Linux supporters say that I am wrong, please show us a link to a 32 socket Linux server. No one has ever showed me such links. Never ever. They claim I am wrong, but... no links. Lot of talking, but no proof. I am wrong, then it is easy to make me shut up: show us a link to single 32 socket Linux server. :)"

            http://www.sgi.com/products/servers/uv/models.html

            256 sockets, single system image coherent shared memory

            http://www-03.ibm.com/systems/power/solutions/open-platform.html?LNK=browse

            http://www-03.ibm.com/systems/power/hardware/795/specs.html

            Linux is now an option for the Power 795, which has a max of 32 sockets.

          3. fch
            Headmaster

            Re: Switching from big iron to x86 virtualisation

            I apologize to the readers for not having given the proper icon first way round. After all, the orig comment I replied to was about 32 CPUs, not 32 CPU sockets ... in any case, I agree that "more" of some sort will give you bragging rights amongst a certain audience.

            Highly profitable the high end may still be, but I'd stand by the assertion that the number of sharks in that pond hasn't changed much, while the pond is drying out. And those who dip their feet in there are either very brave, very foolish, or so desperate as to have no other choice.

            1. Kebabbert

              Re: Switching from big iron to x86 virtualisation

              @fch

              "...I apologize to the readers for not having given the proper icon first way round. After all, the orig comment I replied to was about 32 CPUs, not 32 CPU sockets ... in any case, I agree that "more" of some sort will give you bragging rights amongst a certain audience..."

              So, what is the difference between 32 cpu and 32 socket server? In each socket, there is a... cpu? Right? So, shouldnt 32 cpu server be the same as 32 socket server? I agree that the core count might be different, but a a cpu with 16 cores, or a cpu with 4 cores - they are both one single cpu. And they both sit in a single socket. Or do you have another explanation?

              Regarding Linux and 32 socket servers. There are lot of Linux fanboys that claim Linux scales so well, it is the besto, they say. Well, I ask, how do you know that? Can you show me Linux benchmarks on a 32-socket server that proves Linux scales well? And no, they cant. Because there has never been a 32 socket Linux server for sale. So, how do they know that Linux scales well on large servers with 32 sockets? Pure imagination, I guess. And when I ask for benchmarks on a 32 socket server, they call me names, "dumb", "idiot", etc. At that point I know I have won. Because if they do really have proof they would have shown us the links. But they have nothing, so they revert to using harsh language instead. Just like Linus Torvalds himself.

              So fact is: Linux scales very bad on larger SMP alike servers. It has troubles scaling on 8 socket servers, just study some benchmarks. For instance on SAP benchmarks, Linux used higher clocked cpus and faster RAM sticks, and still Solaris got a higher score, because Solaris had better cpu utilization at 99%, whereas Linux had 87% cpu utilization. Linux scales crap on SMP servers, because no kernel developer can make Linux scale well on large SMP servers, because there exist no such server to test Linux on. However, on clusters Linux scales well, and no one denies that.

              Linux scaling on SMP servers: problems scale above 8 cpus, because there are no larger SMP server than 8 socket. So no Linux kernel developer can tailor the Linux kernel for 16 sockets or 24 or 32 sockets. Or 96 sockets, as the Oracle M6 server has. Solaris, AIX and HP-UX developers have had access to large 32 socket servers for decades for testing, so these Unix runs fine on such servers.

              Linux scaling on HPC clusters: very good.

              1. Roo

                Re: Switching from big iron to x86 virtualisation

                "For instance on SAP benchmarks, Linux used higher clocked cpus and faster RAM sticks, and still Solaris got a higher score, because Solaris had better cpu utilization at 99%, whereas Linux had 87% cpu utilization"

                CPU utilisation is a pretty meaningless statistic. What would you rather have 10 transactions per second with 100% CPU utilisation or 1000tps with 87% utilisation ? You can achieve 100% utilisation with a busy loop ffs.

                "And no, they cant. Because there has never been a 32 socket Linux server for sale."

                Repeating a falsehood when the truth has already been pointed out and the facts are easily available to everyone reading the thread is a silly thing to do.

                For the record I think that big machines with single system images are a daft idea because the laws of nature are working against it (distance, propagation delay, cooling, resource contention). I prefer software to work along the lines of a fault-tolerant network of communicating sequential processes.

                Kebabbert you don't seem to care enough about the topic to understand the terminology, let alone the underlying technology.

                1. Kebabbert

                  Re: Switching from big iron to x86 virtualisation

                  @Roo,

                  "...CPU utilisation is a pretty meaningless statistic. What would you rather have 10 transactions per second with 100% CPU utilisation or 1000tps with 87% utilisation ? You can achieve 100% utilisation with a busy loop ffs..."

                  Please read my post again. I am saying that Solaris, using slower hardware than Linux, scored higher on SAP benchmarks. Why is that? At the same time, Solaris had higher cpu utilization and Linux lower. Coincidence? Maybe. The point is, in 8-socket SAP benchmarks, Solaris scored higher than Linux. On equivalent hardware. Both using Opteron cpus, same opteron model, but one clocked at 2.8GHz (Linux) and one clocked at 2.6GHz (Solaris).

                  Cpu utilization is very meaningful - if you run the same benchmarks. If SAP utilizes resources more on one OS, than another OS - is that "meaningless"? Most people dont agree with you.

                  .

                  "...Repeating a falsehood when the truth has already been pointed out and the facts are easily available to everyone reading the thread is a silly thing to do...."

                  Ok, show all us the truth then. It is only a matter of linking to a Linux server SMP alike with 32 sockets. No, IBM Mainframes with 24 sockets running Linux does not count. Neither does a IBM AIX Unix server running Linux. So, please show us a Linux server instead of calling me a liar. Can you, or can you not? If not, then dont call me a liar, because that makes you the liar - you lie about me.

                  The only 32 socket server capable of running Linux today, is a Unix server. The IBM Unix P795 is only a couple of years old (3 years or so), before the IBM P795, where were any Linux server with 32 sockets? No one existed. And still, prior to the P795 the Linux fanboys claimed that Linux scaled extremely well. On what hardware? That is just laughable. I mean, where is the proof? Where are benchmarks? I can never understand how Linux fanboys can go and raving about Linux excellent scalability - but it has never been tested! Well, it has actually been tested once, on the HP Unix server with 64 cpus, and Linux scaled horribly according to HP benchmarks.

                  "The truth has been pointed out" - what truth? Seriously? In what reality do you live? WHERE ARE THE 32 SOCKET LINUX SERVERS??? WHERE???

                  1. Roo

                    Re: Switching from big iron to x86 virtualisation

                    "The truth has been pointed out" - what truth? Seriously? In what reality do you live? WHERE ARE THE 32 SOCKET LINUX SERVERS??? WHERE???"

                    As others have pointed out: "http://www.sgi.com/products/servers/uv/

                    1) Ships with Linux

                    2) Runs a single OS instance (ie: it's not operating as a cluster but as a single machine with shared memory)

                    It really doesn't matter if you wish to ignore it because you can't find a SAP benchmark for it, the rest of the world can see that SGI's UV1000 is a 256 socket server running a stock Linux image that spanks an M9000 running a custom cut of Solaris and a compiler specifically tuned for the machine by a factor of three in SPECrate benchmarks.

                    Please note that so far I have not made any claims about how well Linux scales on 32 sockets, because I haven't actually run any of *my* code 32 socket Linux machines. That said I expect to see in excess of 80% of peak performance (peak = single core figure*core count) on an 8 socket box in my line of work.

                    When I first cut my teeth on dual processor machines over 20 years ago it was not unusual to see multi-socket boxes delivering less than 65% of peak so I think the hardware has progressed nicely. Back then SMP machines were often starved of bandwidth due to contention long wide busses being crippled by skew & line driving problems.

                    1. Kebabbert

                      Re: Switching from big iron to x86 virtualisation

                      Jesus. Again, that SGI UV1000 is a cluster.

                      http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access#NUMA_vs._cluster_computing

                      "One can view NUMA as a tightly coupled form of cluster computing."

                      And, again, there are fundamental differences between a cluster and a SMP server. As kernel developer explains:

                      http://gdamore.blogspot.se/2010/02/scalability-fud.html

                      "[HPC clusters] spend a huge majority of their time in "user" and only a minuscule tiny amount of time in "sys". I'd expect to find very, very few calls to inter-thread synchronization (like mutex locking) in such applications....Consider a massive non-clustered database. (Note that these days many databases are designed for clustered operation.) In this situation, there will be some kind of central coordinator for locking and table access, and such, plus a vast number of I/O operations to storage, and a vast number of hits against common memory. These kinds of systems spend a lot more time doing work in the operating system kernel. This situation is going to exercise the kernel a lot more fully, and give a much truer picture of "kernel scalability"

                      If you really believe such a SGI UV1000 256-socket cluster is going to run a Oracle database for a tenth of the price of a 32 socket SMP server at a higher performance, why dont all large Investment Banks at Wall Street do that? What are you thinking with? Are you thinking at all? Why is there a market for very very expensive 32 socket servers, if a cheap 256-socket SGI cluster can replace 32 socket servers? Seriously? You dont think at all?

                      A question, you never attended some logic courses or so? Never went to university?

                      1. Roo

                        Re: Switching from big iron to x86 virtualisation

                        "If you really believe such a SGI UV1000 256-socket cluster is going to run a Oracle database for a tenth of the price of a 32 socket SMP server at a higher performance,"

                        Err, I don't believe we were even talking about price, and you won't get an idea of the real prices without talking to your friendly local salesman. :)

                        "why dont all large Investment Banks at Wall Street do that?"

                        Actually some do, I have been working with large Investment Banks for a decade or so now.

                        "What are you thinking with? Are you thinking at all? Why is there a market for very very expensive 32 socket servers,"

                        Probably because people like Oracle have their customers by the short and curlies. $legacy_vendor gets more margin if they can convince you that buying one of their new boxes will cost less than a new $legacy_vendor license + $competitor's box. The $legacy_vendor can set the license for the competition's box to cancel out the price difference AND they can maintain a nice fat margin because there is no competition (that's the whole point of legacy lock-in).

                        "if a cheap"

                        I doubt it's cheap.

                        "256-socket SGI"

                        Yes, 192 more sockets than the Oracle boxes you are so keen on.

                        "cluster"

                        It's not a cluster, it runs a *single* instance of the OS against shared memory. A single process can use every single byte of memory in that system. The same is not true of a cluster.

                        " can replace 32 socket servers?"

                        "A question, you never attended some logic courses or so?"

                        A few too many tbh.

                        " Never went to university?"

                        Did that, it was largely a waste of time as far as course content went because I had already spent 18 months living and working with people who designed and built 32bit microprocessors that got stuffed into systems with thousands of cores in them.

                        I think you should be more concerned over your own education given that keep asserting that a ccNUMA system is a Cluster.

                      2. Roo

                        Re: Switching from big iron to x86 virtualisation

                        "One can view NUMA as a tightly coupled form of cluster computing"

                        ... Is not the same as saying that NUMA systems are tightly coupled clusters, it merely implies that they *may* share similar characteristics. I could view elephants as kittens on the grounds that they have four legs, that doesn't mean that elephants are kittens or that kittens have trunks.

                        "Symmetric multiprocessing (SMP) involves a multiprocessor computer hardware and software architecture where two or more identical processors are connected to a single shared main memory" - Wikipedia.

                        Meanwhile the M9000 is still a NUMA box, which you could view as a cluster (if you want to ignore the shared memory which is the whole point of NUMA), but not an SMP system because it doesn't have a single bank of shared memory.

                        With SMP all memory is remote, so you are operating at worst case (but uniform) latency all the time, by contrast with NUMA you get the best possible latency for local accesses and the same worst possible latency for remote accesses as you would have with an SMP box. All the M9000 does is hide the latency from your code by putting your code to sleep and running another thread whenever it has to hit main memory. Xeons achieve a similar trick with HyperThreading, but HT is a bit finer grained and potentially a bit more useful because it works at the level of execution units. YMMV, some HPC shops switch HT off.

                        Your whole argument about how much the Linux kernel sucks at scaling is rests on the following (unsubstantiated) assertion:

                        "HP had their "Big Tux" Linux server, which was the 64 socket Itanium Integrity (or was it Superdome?) server that they compiled Linux to, and Linux had something like ~40% cpu utilization using 64 sockets"

                        1) Big Tux was actually a project (http://welcome.hp.com/country/uk/en/features/05superlinux.html#quickLink2).

                        2) Big Tux kicked off about 7 years ago (hint: a lot has changed in the Linux kernel since then).

                        3) Given the timing I think they would have been using an Itanic 2 based Superdome, the SD2 came after that effort.

                        I'm done. I can't see much point in writing stuff for someone who shows no evidence of being able to read and learn.

                        1. Kebabbert

                          Re: Switching from big iron to x86 virtualisation

                          "...Another point to consider with respect to your assertion that you will get good scaling from unmodified binaries on an M9000 is that it relies on running a huge number of threads to achieve high system throughput at the expense of a big hit in single-thread throughput. Legacy binaries are usually tuned to run on high-single-thread throughput systems, so I would expect to see better scaling and throughput from a system with a high-throughput cores (eg: Xeon) vs low-throughput cores (SPARC Tx)...."

                          Jesus. You are just totally off. Off. You havent understood much. It is the opposite: The M9000 is old, and use the old SPARC64 cpu which has 2 strong threads per core, and low throughput. The new "SPARC Tx" has very high throughput cores, with many threads. Xeon has low throughput and SPARC Tx (T5, etc) are high throughput. Xeon has in comparion: few strong cores and few threads, and SPARC T5 has many threads to boost throughput. This is just the opposite of what you believe.

                          Relatively this holds, if you compare them:

                          Xeon: low throughput because it has few strong threads

                          SPARC64: low throughput because it has few strong threads

                          SPARC Tx (T5): high throughput because it has many weaker threads.

                          You are stating the opposite. Just check the some of the world record benchmarks here, and see that SPARC T5 is SEVERAL times faster at high throughput server work loads (inlcuding SPECIint2006):

                          https://blogs.oracle.com/BestPerf/entry/20130326_sparc_t5_speccpu2006_rate

                          .

                          "....Probably because people like Oracle have their customers by the short and curlies. $legacy_vendor gets more margin if they can convince you that buying one of their new boxes will cost less than a new $legacy_vendor license + $competitor's box. The $legacy_vendor can set the license for the competition's box to cancel out the price difference AND they can maintain a nice fat margin because there is no competition (that's the whole point of legacy lock-in)...."

                          Wrong again. The Oracle database runs on Linux too. In fact, it is mainly developed on Linux I have heard of lately. So it would be very easy to migrate from Oracle database running on a very very expensive 32 socket server, to a cheap 32/64/128 socket Linux SGI cluster running Oracle database. But no one is doing that, why? And Oracle is not famous for cutting prices when they have you locked in, they are expensive. Many companies wants to migrate to other databases because of the high license costs. If you did not know this, I doubt you have worked at Wall Street as you claim.

                          Why dont no one migrate from a very expensive 32 socket Unix server to a cheap SGI cluster - running the same software? No vendor lock in exists, because you migrate from Oracle, to Oracle. Your explanations why noone does this are logically unsound. I tell you the answer: as the kernel developer explained, these cheap Linux clusters can not run large Oracle database configurations which requires SMP servers, and that is why no one migrates from expensive Unix SMP servers to cheap Linux clusters. Because clusters can not handle huge database configurations. The worst case RAM latency is more than 10.000ns in clusters, and performance of a database would grind to a halt.

                          .

                          "...I doubt [SGI cluster] is cheap...."

                          Wrong. Yes, the SGI cluster is cheap. Check the prices. It can in no way compare to 32 socket IBM P595 server used for the TPC-C record, it costed $35 million list price. I am convinced the SGI cluster costs like a few x86 cpus and a fast switch and not much more, or maybe twice that cost. You can buy several SGI clusters for the price of one 32 socket IBM server.

                          .

                          "...[SGI UV1000] it's not a cluster, it runs a *single* instance of the OS against shared memory. A single process can use every single byte of memory in that system. The same is not true of a cluster...."

                          You are wrong again. For instance, the ScaleMP Linux server with 1000s of cores shows the same charasterica: it runs 8192 cores and loads of RAM, and it runs a single Linux kernel image. But, it is actually a cluster. Just because it runs single image does not mean it is not a cluster. It can only run HPC workloads, just like the SGI cluster. Both clusters consists of several smaller nodes, connected to look like one giant server running single image kernel:

                          http://www.theregister.co.uk/2011/09/20/scalemp_supports_amd_opterons/

                          "... vSMP takes multiple physical servers and – using InfiniBand as a backplane interconnect – makes them look like a giant virtual SMP server with a shared memory space....The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit".... "

                          Check the SGI and ScaleMP workloads, they are all HPC workloads. For a reason. No customer runs a large database configuration such as Oracle.

                          .

                          "...With SMP all memory is remote, so you are operating at worst case (but uniform) latency all the time, by contrast with NUMA you get the best possible latency for local accesses and the same worst possible latency for remote accesses as you would have with an SMP box...."

                          True. The Oracle servers are SMP alike, and has very low worst case latency, 500ns or so. In effect you treat it like a true SMP server. The SGI and ScaleMP clusters have latency of 10.000ns or much higher - these must be treated as clusters, and can only run cluster software. That is why all SGI and ScaleMP customers are running HPC workloads.

                          .

                          "...All the M9000 does is hide the latency from your code by putting your code to sleep and running another thread whenever it has to hit main memory. Xeons achieve a similar trick with HyperThreading..."

                          Wrong again. You are mixing the M9000 cpus with SPARC Niagara cpus. The Niagara cpus hide latency by switching to another thread when the cache pipeline stalls, and it has many cores and many threads to be able to achieve a very high throughput. The M9000 has the old SPARC64 cpu, with only 1-2 threads. The old SPARC64 is very similar to older x86 or odler IBM POWER cpus: few cores, 1-2 strong threads. All cpus where constructed like this long ago. Then came the SPARC Niagara and changed everything with many cores and many threads, and now every cpu is similar to Niagara with many cores and many threads: POWER7, Xeon. The SPARC64 is not good at high throughput, it has few strong threads, not many threads.

                          So, you dont know too much about M9000 or SPARC64 cpus. You are mixing them. That might explain why are so off with your knowledge.

                          .

                          "....I'm done. I can't see much point in writing stuff for someone who shows no evidence of being able to read and learn...."

                          I have proved that you are wrong in many(every?) bit of your reasoning. It is you who needs to read and catch up.

                          1. Roo

                            Re: Switching from big iron to x86 virtualisation

                            You are quite correct, I did mistakenly think the M9000 had Niagra cores (should checked, sorry), but that doesn't change the argument with respect to strong single thread performance vs weak single thread performance given how weak those SPARCs are.

                            When does the rest of the world get to see where you get your figures from so we can see rest easy knowing that you haven't simply made up the latency figures in order to shore up your self-contradictory arguments ?

                            1. TheOrqwithVagrant

                              Re: Switching from big iron to x86 virtualisation

                              Kebbabert is taking latency figures from ScaleMP, which provides an "emulated" NUMA machine on top of a cluster and applying those to SGI Altix UV, which is a genuine hardware ccNUMA architecture. This is about as accurate as taking numbers from qemu's SPARC CPU emulation and using it as an argument against actual SPARC cpu performance. For Altix UV, he's off by around three orders of magnitude on the memory latencies. Even the very first generation of Altix back in 2003 had only an 800ns worst case latency, and there's been ten years of progress since.

                              In fact, by repeatedly implying that SGI Altix UV is "really" just an emulated ccNUMA system like ScaleMP, despite all specifications, design documentation and and sales literature on SGI's site, Kebbabert has gone past "being wrong" in this thread, and is now arguably perpetrating business libel against SGI.

                              To provide some actual figures to show how ridiculously off-base Kebbabbert's numbers are, the NUMALink4 interconnect, which was used for the Itanium-based Altixes and is already obsolete, had a latency of 340ns for the first hop, and then an additional 60ns per hop. Altix UV uses NUMALink5 (more than twice as fast), and has a layout that ensures no more than two hops to the most distant memory, meaning worst-case memory latency to the most distant node on the current Altixes can be safely assumed to be better than the _best case_ latency on the M9000 (437ns). The best case latency to local node memory is less than _30ns_ on the Altix - over ten times better than the M9000. If anyone wonders why "classic" UMA SMP architectures are a thing of the past, you need not look much further than these numbers.

                              1. Roo

                                Re: Switching from big iron to x86 virtualisation

                                I try not to make assumptions, but your guesses seem reasonable. 500ns worst case would be pretty handy across 256 sockets, it would be interesting to see how it copes as memory traffic volumes scale up.

              2. fch

                Re: Switching from big iron to x86 virtualisation

                @Kebabbert:

                Not called you any names, really, apologies if you misunderstood me there. Nor have I ever claimed Linux scales to "ludicrous" (in the spaceballs sense) "numbers of CPUs" (measured by whichever gauge). 32 CPU cores on Linux are "pretty much ordinary" these days, and such systems perform "good enough".

                I also agree both that there are workloads which exceed what you can do with "whitebox HW running Linux", as well as there being hardware/software vendors providing systems capable of running such workloads. Interesting where you need it - if you need it. Technically fascinating ? You bet !

                I've merely made observation that the "good enough" frontier has continued to steadily encroach into that terrain. The pond full of "stuff Linux / x86 hw doesn't do [ well ]" hasn't dried out completely yet, altough, neither, have I personally seen signs of the water level there rising.

                1. Roo

                  Re: Switching from big iron to x86 virtualisation

                  With regards to Linux/x86 being "good enough", here are some SPEC rate fp figures of a couple of boxes with 64 sockets in action:

                  SGI Altix UV1000 (SLES 11, tested Aug 2010): Base 6390, Peak 6600

                  Oracle M9000 (Solaris 10, tested Nov 2010): Base 2270, Peak 2550

                  The gap is even wider in the int benchmarks, but the magnitude of the difference surprised me given that those machines were released about the same time.

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Switching from big iron to x86 virtualisation

                    The Oracle Solaris engineers seem to think the Sparc Enterprise servers are NUMA architectures.

                    From the solaris-sparc-white-paper-167888.pdf‎, page 17:

                    ====

                    NUMA Optimization—MPO

                    As systems grow larger, with more processor sockets and more memory, the ability of a processor to access memory becomes more challenging—all processors cannot directly access all memory at the same latency. Multiprocessor systems generally demonstrate some memory locality effects, which means that when a processor requests access to data in memory, that operation will occur with somewhat lower latency if the memory bank is physically close to the requesting processor. Sun SPARC Enterprise servers are designed with a NUMA architecture, enabling processors to directly access some memory at the lowest latency, while accessing the rest of the memory with more latency.

                    Oracle Solaris provides technology that can specifically help applications improve performance on NUMA architectures. Oracle Solaris uses Memory Placement Optimization (MPO) to improve the placement of data across the physical memory of a server, resulting in increased performance. Through MPO, Oracle Solaris works to help ensure that memory is as close as possible to the processors that access it, while still maintaining enough balance within the system. As a result, many database and technical computing applications are able to run considerably faster with MPO.

                    ====

                    Should I trust the Solaris engineers or Kebabbert? Tough call.

                    1. Kebabbert

                      Re: Switching from big iron to x86 virtualisation

                      @Anonymous Coward

                      "...The Oracle Solaris engineers seem to think the Sparc Enterprise servers are NUMA architectures.... Should I trust the Solaris engineers or Kebabbert? Tough call..."

                      Please read my posts again so I dont have to repeat myself all the time. But the heck, let me recap once more.

                      The M9000 with 32 sockets is not a true SMP server, neither is the new Oracle M6 server with 96 sockets. But they act like true SMP servers because of good design. It is very few hops to reach memory cells far away, the worst case latency on M9000 is 500ns and the best case is 100ns. The spread is tight. So in effect you treat the M9000 as a true SMP server. You dont need to reprogram your software, to make sure data is close to the current cpu, etc. No need for this.

                      In contrast, the SGI Altix cluster is a cluster. It has worst case latency of 10.000ns or 70.000ns, I cant remember the number. This means that you can only run some workloads on a HPC cluster. When you program for a cluster, you must make sure that the data is allocated close the current cpu, otherwise the performance will be very very bad. You must reprogram your software if you intend to run on a HPC cluster. For isntance, SMP workloads such as databases does not run on HPC clusters. No one is running large Oracle databases on a SGI cluster. For a reason.

                      However, the Oracle M6 server with 96 sockets and 10.000 threads and 96TB RAM - is specifically designed to run databases and other SMP Enterprise workloads. So, you dont have to redesign your software running on M6 server, just copy your binary to the M6 server without problems. In effect the M6 server is a SMP server.

                      Did you understand why the Oracle M9000 (which is a non SMP server) behaves like a SMP server in real life? You dont have to redesign all your software. But for a HPC cluster, you must.

                  2. Kebabbert

                    Re: Switching from big iron to x86 virtualisation

                    @Roo,

                    The SGI Altix UV1000 cluster sure beats any SMP servers on HPC number crunching workloads. The SGI Altix cluster was made for it. I would be very surprised if a Enterprise SMP server such as the Oracle M9000 beat a pure number crunching cluster in number crunching.

                    On the other hand, if you benchmarked the SGI cluster on SMP workloads, you would see very bad performance. The M9000 is made for SMP, and SGI made for HPC. I dont know how many times I have to say this? They are running different workloads, one is running number crunching, the other is doing SMP. The SMP can run trivially parallelisable problems such as number crunching, but a HPC can not run SMP workloads with latency of 10.000ns to 70.000ns - that would be impossibly slow. That is the reason no one on Wall Street use HPC servers for databases and other Enterprise workloads.

                2. Kebabbert

                  Re: Switching from big iron to x86 virtualisation

                  @fch,

                  I was trying to explain to you, how the Linux fanboys are wrong when they talk about Linux supreme scalability. I am not meaning that you are like them. Sorry if I was unclear.

                  But I hope you see my point. There have never been any Linux 32 socket server for sale so the Linux kernel developers could not have improved Linux for 32 socket scaling. You need to test your code on live hardware, and such hardware does not exist. Sure, some people have tried to compile Linux onto Unix and Mainframes, but just because it works badly and scales badly, does not make them good options. There is no way in hell that Linux can scale well on 32 sockets, because there are no such benchmarks.

                  Well, actually, HP did benchmark once on 64 sockets (Big Tux server), and Linux sucked badly with ~40% cpu utilization. Of these 64 cpus, around 26 cpus were used and the rest 38 cpus were idling. More than half of the cpus were just sitting there, rolling their thumbs - under full load. That is bad actually. Very bad. You want to use all cpus and use all resources in a server. It can only take a deluded Linux fanboy like "Roo" to think that is a good result. 40 cpus idling on a 64 cpu server under full load - is that good scaling???

        2. Roo

          Re: Switching from big iron to x86 virtualisation

          "Yes, there's not that many x86 servers out there that can have 32 CPU sockets. If that's what counts for you, go IBM / Fujitsu / Oracle."

          If your definition of x86 servers can stretch (a lot) the Cray XC-30 might be of interest. It ships with the Cray Linux Environment, a cabinet can hold 384 sockets (3072 Xeon E5 cores), infinband I/O. You can add a lot of cabinets too. At the lower end you have the SeaMicro boxes ranging from 64 to 256 sockets (and I have seen them referred to as 'servers').

          There is a lot of choice at the moment if you are prepared to look beyond the usual suspects, and the choice will get broader very quickly if Chinese vendors get some traction.

      4. Vociferous

        Re: Switching from big iron to x86 virtualisation

        Just curious what you consider to be "enterprise work load"?

        I can envision situations where Linux would not be my first choice, but I can't think of any situation where Windows would be an option but Linux wouldn't - and that's the situation implied in the article.

      5. JEDIDIAH

        Re: Switching from big iron to x86 virtualisation

        > But there are no high end SMP Linux servers for sale and has never been,

        The largest SMP server was a Linux server sold by SGI.

        Also, large workloads may very well do better with some form of clustering which marginalizes the sort of large NUMA box you seem to be fixating on. This applies equally well to commercial Unix as it does Linux.

        Also, a big Unix server is more likely to be more akin to a virtualization solution than something required for a "big problem".

Page:

This topic is closed for new posts.

Other stories you might like