back to article Linux in 2016 catches up to Solaris from 2004

In 2004 former Reg hack Ashlee Vance brought us news of DTrace, a handy addition to Solaris 10 that “gives administrators thousands upon thousands of ways to check on a system's performance and then tweak ….production boxes with minimal system impact”. Vance was excited about the code because “it can help fix problems from the …

  1. Voland's right hand Silver badge

    BPF is not the only way to get a timed dump

    Linux has had ring buffer capture for ages and it is actually supported by libpcap. That has timestamp-at-capture as well as support for hardware assisted timestamp (the latter - if you have the appropriate hardware). In fact, this exact "feature" has forced me to use raw instead of bpf/pcap on a couple of occasions in the past. Timestamping is an extremely expensive operation - you DO NOT WANT that enabled unless you really, really need it. It costs quite a bit in terms of performance and it also forces lots of cache synchronizations so chasing down race conditions and/or broken locking becomes more difficult.

    I do not quite see how this relates to dtrace though - dtrace can trace anything while bpf (or in Linux case to be more exact - lpf) is used to perform matching on a stream (usually packets, though linux also uses the same bytecode generator for security policies, etc).

    1. Tomato42
      Boffin

      Re: BPF is not the only way to get a timed dump

      bpf stopped to be a stream matcher good few releases ago, it's a generic system that just happens to be the same thing you use for packet capture

  2. PlinkerTind

    Slowly, slowly

    Linux is getting the tech from Solaris: ZFS, DTrace, SMF (systemd), Containers (dockers), etc etc etc. 10 years later. Better later than never, though.

    1. Jad

      Re: Slowly, slowly

      While most of what you've said is true, dockers are not containers, they are completely different things that appear on the outside to look similar. containers allow resource limitation on a scale that dockers don't ...

      And while systemd works, I recall (I can't find the references) that there was a linux guy who reviewed the code from OpenSolaris and referred to it as a "battleship" compared to our "rowing boat" ... (he went on to say that he didn't think that Linux needed or wanted a battleship)

      Oracle were porting Dtrace to their Oracle Solaris, but I don't believe they have / certain that they haven't released it to the wider community.

      That said, Linux USB support is much better and more resilient, and support for hardware means that I can run it on this laptop ...

    2. iOS6 user

      Re: Slowly, slowly

      > Linux is getting the tech from Solaris: ZFS, DTrace, SMF (systemd), Containers (dockers), etc etc etc

      ZFS on Linux bases on OpenZFS which is years behind of what Solaris ZFS now provides.

      btrfs in latest kernel 4.9rc2 in first few seconds after mounting root fs OOPes on my laptop with detecting recursive locking:

      [ 19.847017] =============================================

      [ 19.848461] [ INFO: possible recursive locking detected ]

      [ 19.849904] 4.9.0-0.rc2.git2.1.fc26.x86_64 #1 Not tainted

      [ 19.851360] ---------------------------------------------

      [ 19.852823] systemd-journal/491 is trying to acquire lock:

      [ 19.854283] (

      [ 19.854298] &ei->log_mutex

      [ 19.855740] ){+.+...}

      [ 19.855751] , at:

      [ 19.857234] [<ffffffffc0594372>] btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.858627]

      but task is already holding lock:

      [ 19.861547] (

      [ 19.861562] &ei->log_mutex

      [ 19.863042] ){+.+...}

      [ 19.863053] , at:

      [ 19.864566] [<ffffffffc0594372>] btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.866079]

      other info that might help us debug this:

      [ 19.869067] Possible unsafe locking scenario:

      [ 19.871622] CPU0

      [ 19.873098] ----

      [ 19.874558] lock(

      [ 19.874575] &ei->log_mutex

      [ 19.876023] );

      [ 19.877432] lock(

      [ 19.877449] &ei->log_mutex

      [ 19.878900] );

      [ 19.880319]

      *** DEADLOCK ***

      [ 19.884367] May be due to missing lock nesting notation

      [ 19.887001] 3 locks held by systemd-journal/491:

      [ 19.888316] #0:

      [ 19.888332] (

      [ 19.889627] &sb->s_type->i_mutex_key

      [ 19.889644] #12

      [ 19.890938] ){+.+.+.}

      [ 19.890948] , at:

      [ 19.892278] [<ffffffffc0562da3>] btrfs_sync_file+0x163/0x4c0 [btrfs]

      [ 19.893624] #1:

      [ 19.893640] (

      [ 19.894959] sb_internal

      [ 19.894970] ){.+.+.+}

      [ 19.896274] , at:

      [ 19.896307] [<ffffffffc05499f6>] start_transaction+0x2f6/0x530 [btrfs]

      [ 19.897660] #2:

      [ 19.897676] (

      [ 19.899034] &ei->log_mutex

      [ 19.899046] ){+.+...}

      [ 19.900396] , at:

      [ 19.900431] [<ffffffffc0594372>] btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.901819]

      stack backtrace:

      [ 19.904602] CPU: 2 PID: 491 Comm: systemd-journal Not tainted 4.9.0-0.rc2.git2.1.fc26.x86_64 #1

      [ 19.906037] Hardware name: Sony Corporation VPCSB2M9E/VAIO, BIOS R2087H4 06/15/2012

      [ 19.907493] ffffaf42c165b820 ffffffffb746cb43 ffffffffb8be9350 ffff9ff58ada0000

      [ 19.908930] ffffaf42c165b8e8 ffffffffb7111cbe 0000000000000002 ffffffff00000003

      [ 19.910363] 00000000ca125543 ffffffffb84bf600 a2c5096b94a6fc27 ffff9ff58ada0ca8

      [ 19.911768] Call Trace:

      [ 19.913071] [<ffffffffb746cb43>] dump_stack+0x86/0xc3

      [ 19.914393] [<ffffffffb7111cbe>] __lock_acquire+0x78e/0x1290

      [ 19.915713] [<ffffffffb70ece67>] ? sched_clock_cpu+0xa7/0xc0

      [ 19.917019] [<ffffffffb7907a5e>] ? mutex_unlock+0xe/0x10

      [ 19.918318] [<ffffffffb7112c26>] lock_acquire+0xf6/0x1f0

      [ 19.919662] [<ffffffffc0594372>] ? btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.920966] [<ffffffffb7906de6>] mutex_lock_nested+0x86/0x3f0

      [ 19.922289] [<ffffffffc0594372>] ? btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.923613] [<ffffffffc05aaa85>] ? __btrfs_release_delayed_node+0x75/0x1c0 [btrfs]

      [ 19.924936] [<ffffffffc0594372>] ? btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.926268] [<ffffffffc05ac919>] ? btrfs_commit_inode_delayed_inode+0xe9/0x130 [btrfs]

      [ 19.927620] [<ffffffffc0594372>] btrfs_log_inode+0x162/0x1190 [btrfs]

      [ 19.928951] [<ffffffffb70e0f8a>] ? __might_sleep+0x4a/0x80

      [ 19.930290] [<ffffffffc0594f28>] btrfs_log_inode+0xd18/0x1190 [btrfs]

      [ 19.931607] [<ffffffffb7037de9>] ? sched_clock+0x9/0x10

      [ 19.932940] [<ffffffffc0595670>] btrfs_log_inode_parent+0x240/0x940 [btrfs]

      [ 19.934264] [<ffffffffb72c6279>] ? dget_parent+0x99/0x2a0

      [ 19.935628] [<ffffffffc0596d52>] btrfs_log_dentry_safe+0x62/0x80 [btrfs]

      [ 19.937010] [<ffffffffc0562f52>] btrfs_sync_file+0x312/0x4c0 [btrfs]

      [ 19.938385] [<ffffffffb72e70eb>] vfs_fsync_range+0x4b/0xb0

      [ 19.939768] [<ffffffffb72e71ad>] do_fsync+0x3d/0x70

      [ 19.941092] [<ffffffffb72e7470>] SyS_fsync+0x10/0x20

      [ 19.942251] [<ffffffffb7003eec>] do_syscall_64+0x6c/0x1f0

      [ 19.943665] [<ffffffffb790b949>] entry_SYSCALL64_slow_path+0x25/0x25

      Problem started about two months ago https://bugzilla.redhat.com/show_bug.cgi?id=1366869 and seems no one in Linux community is interested sorting it out.

      Nevertheless btrfs is not even close to ZFS because it does not uses free lists.

      DTrace ond OBP?

      Comparing those two is kind of a joke.

      - Injected OBP code does not have basic sanity checks so it is quite easy hang whole system

      - There is no clearly defined providers so forget about building library of OBP scripts which will be possible to use for next few years

      - namespace of trace points is messy. Every subsystem has own convention. Sometimes tracepoint names ends on begin/end, sometimes on enter/exit and sometimes begin/end is in the middle. Looks like no one controls this and refuses some some random new naming conventions. Some people started forming hierarchy in providers names with slashes :-0

      # perf list | awk -F: '{print $1}' | sort | uniq | wc -l

      160

      So this is number of kind of perf providers.

      And everything on top of:

      # perf list | wc -l

      1743

      only a bit less than 2k tracepoint when on Typical Solaris now are available 80-90k.

      - SMV vs systemd?

      On Solaris adding SMF not caused any changes in init. On Linux systemd does theoretically everything from init things to crond, PM, logging, login user session and few other things.

      None of the even Fedora rawhide systemd services are ready to create so easy instance of the services like using SMF on Solaris.

      Linux problem is constantly thes same and is called NIH (Not Invented Here) which is causing that initial impulse to implement something new which is already working well on other OS is causing that instead implementing some base functionalities most of the people are attracted on implementing whistles and bells.

      More than 10 years after Solaris 10 release Linux still is deep in the wood.

      Containers and docker?

      Solaris has rock solid non-global zones and has integrated kernel-zones. Linux has Xen and kvm but both of them are not even close to K-Z from point of view for example network layer overhead.

      Solaris started provide well isolation of the processes on top of SMF. Linux SElinux here still is useless because it provides only global AVLs without possibility to cage process on top of service definition.

      Docker is now integrated in Solaris as well.

      In other words .. Linux still is a bit better than only toy which can blow up in your face almost at any time by New-Briliant-Idea-Of-Doing-Old--Things-New-Way.

  3. Alan Sharkey

    There's still nothing as good as ..

    Clustering in OpenVMS. Now, when Linux gets that, I'll consider it has reached maturity.

    1. Anonymous Coward
      Anonymous Coward

      Re: There's still nothing as good as ..

      C̶l̶u̶s̶t̶e̶r̶i̶n̶g̶ ̶i̶n̶ OpenVMS. Now, when Linux gets that, I'll consider it has reached maturity.

      FTFY

      1. Alan Sharkey

        Re: There's still nothing as good as ..

        Of course - fix approved :)

    2. Wayne Sheddan
      Pint

      Re: There's still nothing as good as ..

      For me its SYS$TRNLNM and everything that function call means..... especially if you take the time to play with LNM$FILE_DEV.

      Unix environment variables - hardly transparent to applications. What a hack they are!

  4. Alfred

    About Brendan Gregg

    As an aside, Gregg's presentations and helpful webpages about profiling on Linux are gold. The useful information to noise ratio is very high, and just a few hours spent paying attention to him imparts more than enough practical information to really make a difference when performance tuning your own code. At risk of sounding like his agent, he's worth listening to if you want to know how to profile your software (and its interaction with your hardware) on Linux.

    1. John Riddoch

      Re: About Brendan Gregg

      Even if you don't understand DTrace, the DTrace toolkit Brendan provided is excellent and for most admins does over 75% of what you'd want to do with DTrace in a bunch of easy scripts.

      I've done some basic stuff with DTrace over the years and the power to trace what is going with negligible performance impact is amazing; it's found the source of a few issues over the years (mainly proving the problem was the application...), here's hoping the Linux equivalent gets the care that DTrace has had.

  5. John Sanders
    Trollface

    I'm wondering...

    "Linux in 2016 catches up to Solaris from 2004"

    Shouldn't the question be:

    Has Solaris 2004 catched up with Linux yet?

    Last time I checked one had to install all the GNU utils and Gnome on Solaris if the experience was to be decent.

    Penguins, penguins everywhere, the penguins are coming! the penguins are coming!

    1. wikkity

      Re: GNU utils

      have nothing to do with Linux. Linux is an OS, the gnu tools are user software. The user decides what software they like to use on the OS of their choice, if they decide the gnu toolchain is better that bundled with the OS, that should not reflect on the OS itself.

      1. John Sanders
        Trollface

        Re: GNU utils

        I dunno... Troll face not a clue as to the comment being a joke?

        1. Hero Protagonist

          Re: GNU utils

          "I dunno... Troll face not a clue as to the comment being a joke?"

          Write better jokes.

          1. John Sanders
            Happy

            Re: GNU utils

            Judging by the number of down-votes I'd adventure to say; Mission accomplished.

      2. John Sanders
        Holmes

        Re: GNU utils

        ""if they decide the gnu toolchain is better that bundled with the OS, that should not reflect on the OS itself.""

        I understand what you say, however the OS is both kernel drivers and system libraries plus user-space tools.

        One can't say the GNU tools aren't a large part of any of the shelve Linux OS, this gets to the point to make the distinction pointless to most people. The fact that the GNU tools are built around posix and can be ported to any posix compliant OS does not detract from this. You will struggle to find a Linux OS distro without the GNU tools, while the other way around is common.

        That's why Stallman insists on calling it GNU/Linux

        Just a thought, not trying to be polemic.

    2. Anonymous Coward
      Anonymous Coward

      Re: I'm wondering...

      >one had to install all the GNU utils and Gnome on Solaris if the experience was to be decent.

      Trolling aside. Gnome by the way comes with Solaris 11 as the default WM (Gnome 2, none of this Gnome 3 Linux only garbage Red Hat is foisting on the world). But to each his own. If someone wants to increase the attack surface of their Solaris box by loading spaghetti hairball GNU code on there you are perfectly free too. POSIX is great while it lasts (until Red Hat kills it).

  6. captain_solo

    Dtrace is only one example of how this is true.

    I remember the first formal admin classes I had on a Linux variant (after using it for several years) and my impression at that time was that Red Hat specifically was 3-4 releases behind Solaris in OS features and basic functionality that they were trumpeting as being the greatest ever had been standards in Solaris for years. Especially when dealing with the kernel, I was like "is this Solaris 2.6?" It has gotten somewhat better since then.

    The difference in the feel to me is that for the most part, you can tell that tons of hours and money were spent in the development of Solaris for the benefit of operators and resiliency. Linux still feels a little more loosely bolted together to me. Oh, you want that feature, well just install this sort of compatible open source utility, etc. Generally that is still available in Solaris if needed, but required much less often because more functionality was already integrated into the OS/Kernel

    Both are very good choices with different but overlapping use cases. I am really glad to see many of the most useful innovations moving from Solaris to Linux in various forms.

    1. asdf

      Its all about the VMs and Solaris sadly runs very poorly if at all with less than 2 gig of memory (especially with ZFS). That is the one big use case Linux has over Solaris and easy to poo poo but its a bigger use case than one would think.

    2. Anonymous Coward
      Anonymous Coward

      Both are very good choices with different but overlapping use cases. I am really glad to see many of the most useful innovations moving from Solaris to Linux in various forms.

      You can count me as well. Instrumentation is heavily used here as most of the screwy things I do need fine tuning before I drpo it into the wild. Especially hybrid sets, by which I only count Windows some of the time.

  7. klaxhu

    it is fun to see how people say solaris and openvms are so much better and had this and that 10 years ago, yet the reality is that openvms is basically dead and solaris is on its way out even with oracle at helm

    in terms of adoption, net new servers sold with it etc, linux is way over all the rest combined. don't even get me started with aix, hp ux, etc I have never in a year since I work in IT have heard so many customers they want to move off unix and your typical oracle/sql db's over to better alternatives

    1. Anonymous Coward
      Anonymous Coward

      True but

      The story of computing is the story of "good enough" being the horse you want to bet on in the long run over the "superior" challenger.

      OSI vs TCP/IP;Windows NT over various UNIX; Linux over Solaris, HP-UX; UNIX over VMS. (Mainframes are a curious exception but that is I suspect because they have done an absolutely stellar job of co-opting new movements into their universe. You've been able to run Linux on z/OS for what, 20 years?)

      This trend is exacerbated by cloud computing models where you explicitly reject the "pets" model of computing and treat all your instances as cattle. Once you go down that path, all the robustness advantages of Solaris, HP-UX etc count for very little.

      I will always have a soft spot for Solaris, which was a seriously good OS produced by engineers who really knew their stuff. And (back in the day) supported by great folks whose first response was not to turn it off and then on again. But the future is unquestionably Linux. The only questions now are how quickly it can get all the goodies from the past, and whether the edifice will collapse under its own weight on those somewhat shaky foundations.

      1. asdf

        Re: True but

        >I will always have a soft spot for Solaris, which was a seriously good OS produced by engineers who really knew their stuff.

        Definitely not the same SUN crew that did their JRE implementation. That thing was total shit for at least the first ten years of its life.

      2. Gerhard den Hollander

        Re: True but but hardware

        I've had a softspot for Sun ever since I saw the knightrider lights on the 3/50. Unfortunately in the late 90s sun lost it when their top of the line videocards were being outperformed by a ridiculous margin by a linux box + top of the line NVidia card. And the linux+videocard cost a third of what the sun videocard alone cost. And when in early 2000 the amd and Intel CPUs started outperforming the Solaris machines by similar margins (at least for the software we developed) it became a matter of years before all our clients had switched to linux.

        Sad. But if a 60K machines takes a month to run a job, and a pc you buy for 2k at the shop around the corner runs the job in 2 weeks you have lost.

        Esp since installing linux, loading our software and loading the data took less then a day.

      3. Anonymous Coward
        Holmes

        Re: True but

        Maybe on the Linux future. I'm seeing enough new hardware architecture changes that, put together well, may require a completely new software stack as well. Especially the kernel where assumptions are wired in and optimized for a particular architecture. Sill a ways out, and it's highly unlikely that I'll live to see it. (5 years, give or take.)

      4. DrXym

        Re: True but

        "The story of computing is the story of "good enough" being the horse you want to bet on in the long run over the "superior" challenger."

        I think in this case it's more that Linux comes in a variety of pay models (including free), supports, runs and scales onto any hardware you have, has a better desktop experience, has many more tools, better virtualization, better community support etc. So "good enough" often means superior.

        I'm sure there are gaps where the commercial OS wins such as dtrace but unless that's your #1 requirement is dtrace, it probably doesn't matter enough to decide in its favour. The irony is that even Oracle seem more interested in Linux these days and the release cycles for Solaris are getting ever further apart.

      5. John Sanders
        Linux

        Re: True but

        """whether the edifice will collapse under its own weight on those somewhat shaky foundations."""

        Remember, Linux wasn't meant to be big.

        As hardware evolves so will Linux, video in Linux was traditionally a walking disaster, nowadays it is far from perfect, but in the space of a year it now allows one to play games at 1920x1080 with moderate fps on any recent card, I predict 120fps next year instead of the 40fps we have now.

        Linux has a tendency to snowball every now and then, suddenly some sub-system out of the blue gets much better.

        This is a tendency on most open source software, it's the power of having the source code.

        The code for Solaris... well, we do not have it, Larry does.

  8. Frank N. Stein

    Oracle

    Since Oracle owns Sun Microsystems and thus, Solaris, is Oracle going to sue?

    1. TVU Silver badge

      Re: Oracle

      "Since Oracle owns Sun Microsystems and thus, Solaris, is Oracle going to sue?"

      That is a fair question to ask and DTrace is covered by a Common Development and Distribution Licence. Although it's not as pure as a GNU public licence, it's still an open source licence and it would be an uphill struggle even for litigious Larry at Oracle to try to convince a court that it really ought to be proprietary software.

      So far there's been no challenge on the Ubuntu/ZFS issue and I don't expect one either in respect of DTrace.

      1. Anonymous Coward
        Anonymous Coward

        Re: Oracle

        "No one expects the..." Leisure Suit Larry.

  9. JustNiz

    so that means in 2031

    Windows will get it.

    1. Anonymous Coward
      Anonymous Coward

      Re: so that means in 2031

      Windows already haves the performance monitor which is using graphs so much more information and easy to see than you will have on the linux ascii.

      And also for to use this new linux you will need to know name of programmers dog or something or you won't know what his new command name is.

      1. GrumpenKraut
        Facepalm

        Re: so that means in 2031

        > ...or you won't know what his new command name is.

        Unlike in exactly which O/S?

        1. Bob Gateaux

          Re: so that means in 2031

          > ... Unlike in exactly which O/S?

          OS/400 for one.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like