back to article Linux kernel's Kroah-Hartman: We're not struggling to get new coders, it's code review that's the bottleneck

Speaking in an "Ask the Experts" session at the online Open Source Summit Europe conference today, Linux kernel's stable branch maintainer Greg Kroah-Hartman said there are plenty of new contributors to the code though the bottleneck is finding the people to review it. Perhaps in response to comments reported here from Linux …

  1. Chewi
    Thumb Up

    He's right

    I can totally echo that. As a Gentoo developer, I get new pull requests on GitHub almost daily. I can barely find enough time to make my own changes these days, never mind review other people's. I do try though. I just wish they'd go that extra few miles and actually become bona fide Gentoo developers, then they can take responsibility for their own changes instead of getting stuck behind the bottleneck that is me.

  2. Claverhouse Silver badge
    Holmes

    Retards...

    Although not totalitarian, I wouldn't mind private companies like Intel having to give over 5% of their shares to the injured party every time they were stupidly irresponsible and/or malicious with putative consequences.

    1. gobaskof

      Re: Retards...

      Perhaps even 5% of the value of their shares as of the date the rule was made? Because with such a law in place the value might go down rapidly, as the company bleeds away to others.

    2. Anonymous Coward
      Anonymous Coward

      Re: Retards...

      5% would likely lead to a significant decline in share value resulting in companies breaching their banking covenants and defaulting on loans.

      Cue end of company and lots of "we didn't think this through did we"...

      Surely the injured parties should only be allowed to reclaim the value of the products they purchased upto a multiple of the purchase price.

      1. teknopaul

        Re: Retards...

        maybe, it might also lead to companies not making the mistakes.

        It might lead to a world where "too big to fail" stops getting used to justify sistemic failure.

  3. Lorribot

    This article highlights a lot of the problems there are with the collaborative development model.

    Even Linux has issues with code validation and with many other parties developing parts of the Linux OS with their own needs and wants all pulling at the central core with their own interpretation of the rules.

    What would happen to Linux without Linus as ringmaster?

    "We fix known security issues every single week. We fix tons of unknown security issues every single week." So same problems that Windows has then just keep it quiet then.

    With Linux being used in ever more control systems for things like safety critical environments and likely large infrastructure (not forgetting most of the world's websites) the likelihood of it being aggressively targeted by hackers (both state funded and criminal) is a certainty and then we shall see how well they Linux ecosystem is being managed by all those Application stack DevOps types that have been focused on delivering an app rather than delivering to be managed.

    I wonder if Linux will use the same patch Tuesday?

    1. _LC_
      Holmes

      Linux is a bad example

      Linux is a really bad example, as it's a monolithic Moloch written in C. This turns collaboration into a game of Mikado. ;-)

      1. Anonymous Coward
        Anonymous Coward

        Re: Linux is a bad example

        Go back to your javascript kiddie. You have no idea what you're talking about.

    2. Glen Turner 666

      "Linux has issues with code validation" isn't correct. It is clear where every line of code comes from.

      "What would happen without Linus as ringmaster". Linux *has* a succession plan, at the moment that is GK-H. The risk is greater for other OSs: can you tell me who the successor to Microsoft COSINE's Jason Zander is as the engineering leadership for Windows? That's likely to be determined by business and poltiical issues at Microsoft at the time Zander steps down from that role. The same is true for Apple's engineering leadership. I don't know why you demand a different standard for Linux's engineering leadership.

      As for resolving competing priorities, the lesson of Linux is that operating systems win by addressing everyone's needs. For example, it turns out that everyone wins if the kernel is capable of realtime scheduling, even if they aren't running a realtime application. Similarly, those small realtime systems win by using filesystems with features initially designed for large enterprises in mind.

      "the likelihood of it being aggressively targeted by hackers (both state funded and criminal) is a certainty". Well yes. Because it is has already happened. But you are mistaken that the problem is "all those Application stack DevOps types". The DevOps technologies -- at their heart: easy safe continuous deployment -- makes preventing and responding to security issues much faster. The stronger isolation of Docker and similar technologies is also a win in limiting the fallout from security compromises. The security track record of real world deployment of this technology -- notably in the Google and Microsoft clouds -- is impressive.

      1. Doctor Syntax Silver badge

        "The DevOps technologies -- at their heart: easy safe continuous deployment"

        What that seems to facilitate is "Oh, there's a problem. We'll fix it in the net a later release."

        Personally I preferred getting it right, then releasing it.

        1. teknopaul

          "I do testing so I don't have bugs" has been proven time and again to be a dangerous simplification of the realities of writing code.

        2. admiraljkb

          "Personally I preferred getting it right, then releasing it."

          Regardless of methodology used, Software doesn't release, it escapes as perfect code doesn't exist. Agile dev done properly means less flaws releasing, and much faster fixes when flaws are found. (It is that "done properly" bit that's the kicker though)

          The other thing I've noticed is many big bang release shops do the same thing on fixing issues. They get to it when management makes it a priority, and not a minute before.

          1. _LC_

            "...as perfect code doesn't exist."

            I can't hear this lame excuse for bad coding anymore.

            Only recently I watched a video of someone fixing a bug in a Linux driver. This bug was eons old and introduced, despite people reporting that it broke things, despite it not fixing anything. It boild down to an ugly macro, which didn't do anything. Due to its name, people thought it would do something and wrongly used it for that purpose (endianess). In order to find out what that macro did, or rather did not, you had to walk through tons of #ifdefs and pick your favorite one. This is the shit-show of Linux code we are facing.

            1. oldcoder

              Re: "...as perfect code doesn't exist."

              No, that is people ignoring Linux practice.

              Linus does not like ifdefs with complex structures.

              1. _LC_

                Re: "...as perfect code doesn't exist."

                Did anyone mention structures?

  4. gerryg

    Unfortunately "Torvalds as ringmaster" trope not dead yet.

    I thought this had been killed off about 10 years ago. Linux has not been produced in his bedroom for some time now. While he is still project lead it is the army of lieutenants that do the brunt of it all.

    1. Doctor Syntax Silver badge

      Re: Unfortunately "Torvalds as ringmaster" trope not dead yet.

      It just goes to show how indomitable the human will is in the face of facts.

  5. Anonymous Coward
    Facepalm

    Linux and more

    It's not just the kernel, or Linux as a whole. It's development in general.

    Coding is fun. It's why we got into this. Reviewing, documenting, or managing code isn't.

    The open source community has an added problem with its reliance on volunteers but even in corporate IT, developers who are assigned to review code often lack the focus they have when they are assigned to write code.

    1. Notas Badoff

      Re: Linux and more

      "My PR was accepted! I fixed my problem! I have no more problems. Done!"

      (Also, now I have the needed OSS 'token' I can point to in my next review / job interview.)

      Me, I read code. I read code because at least half the time I start screaming "this is totally wrong and can't work" I find out that coder knew something I'm just now learning correctly, upon trying to verify the details of the wrong thing to include in my snide evaluation.

      Boring reviews are boring, yes. Educational reviews are priceless for everybody.

    2. chuBb.

      Re: Linux and more

      Docs and tests just ain't sexy to write.

      New hires find out quickly that if they fail to cover > 60% of functional logic with tests AND at least provide line by line commentary with comments then they keep at it until it does. Once they have settled in and allowed near anything with a public interface the requirements are stricter.

      They don't like it at the time, but yet to be called a dick 18months later when they are having to maintain there legacy...

      1. gobaskof

        Re: Linux and more

        Line by line commentary gets ridiculous especially because people often comment what they are doing and not why. If you just just unleash someone and demand line by line comments you get wonderfull code that looks like this:

        # setting i to 1

        i = 1

        # looping from 0 to 9

        for j in range(10):

        #adding j to i

        i += j

        There are lots of comments, they offer 2/3 of fuck all insight though.

        1. DJV Silver badge

          Re: Linux and more

          Indeed!

          The "best" one I ever came across was a whole bunch of impenetrable and almost undocumented Perl. There was just a single comment against one piece of the code, which "very helpfully" announced:

          // This is a skanky hack!

          1. John Sturdy
            Coat

            Re: Linux and more

            My favourite, in code originally written by someone with a reputation for assuming others would immediately understand his thinking, and then pored over by others who took some time to plumb the twisted depths, was something like:

            /* A comment here would be helpful. If you can understand it without one, you need help. */

            I think the code concerned used an indexing scheme that started at -2.

          2. maffski

            Re: Linux and more

            // This is a skanky hack!

            That strikes me as a very useful comment - a warning that you should think about it rather than read it.

            The ones that annoy me are things like:

            // Check that the customer has sufficient balance to place the order

            That shouldn't be described by a comment. It should be described by a method name.

            1. Doctor Syntax Silver badge

              Re: Linux and more

              Agreed that things such as naming conventions etc should make code as self-documenting as possible. The comment should not need to tell you that this is a check on customer balance but it might need to tell you that this is the company standard code to be used everywhere such a check is needed; online ordering, telephone ordering or whatever.

              Other things: copyright terms for open source code, why we initialise to 1 or 0 or index from -2*, why we took this approach rather than some other or the fact that this code deals with stuff covered by regulatory requirements and changes should be discussed with and signed off by the appropriate officer of the company.

              * I've used indexing from 400 to 700. They were wavelengths in nm for various data points and Pascal allowed such arbitrary indexes.

          3. Roger Kynaston

            Re: Linux and more

            I think it was in Solaris 2.6 - and possibly as far as 9. The startup script for inetd had the comment in it

            # why oh why did we mess with inetd!

        2. Brewster's Angle Grinder Silver badge

          Re: Linux and more

          I used to write comments like that - when writing asm. Every statement would have a comment explaining what was going on in something that looked more like a conventional higher-level language. Then I started programming C and it all seemed a bit pointless.

          I'll second the code should tell you what; the comments why, although a brief summary of every function or method is a worthwhile introduction. And don't forget assertions - they can tell you useful truths about code; if the state is non-intuitive, asserting it can underline that point.

          1. teknopaul

            Re: Linux and more

            Best feature of rust, code and tests with asserts in the same file.

          2. Claptrap314 Silver badge

            Re: Linux and more

            I got so fed up chasing down overwritten variables that I created a preprocessor that was also a checker. Which meant that I had to declare my variables. By using useful names, comments became limited to the tops of functions. The source code shrank 20%. And it all became MUCH easier to maintain.

        3. teknopaul

          Re: Linux and more

          I call them SBO comments, "Stating the Bleeding Obvious", this works for me as a rule of thumb for what, and what not, to comment.

        4. chuBb.

          Re: Linux and more

          Yeah it gets sent it back if you are documenting syntax and not functional logic same with tests that test language/framework behaviour.

          Sort of things i expect as a minimum is plain speech statements of logical intent like "//check if its the 3rd thursday of the month" (preferably inline and not as a multiline comment out of context), all method args and return types documented, and in the case of maintenance to code a note of why the change came to be, "//new feature, needs to do this, //3rd party API changed" etc.

          I just dont tell new hires this from the get go, as they should have spent long enough looking at our existing code at this point to notice all production code is written like this, somewhat of an attitude/aptitude test i admit, but does give me a good idea of how much mentoring they may or may not require, and how rigidly they define them selves as a coder vs developer....

          1. Brewster's Angle Grinder Silver badge
            Facepalm

            Re: Linux and more

            "..same with tests that test language/framework behaviour."

            I saw a blog where the author took some run of the mill maths and tested it against every single floating point bit pattern. I think it was single precision, but, still, testing whether each of (2**24) - 2 different NaNs behaves the same is a hardware test. I know floats can be tricky. But...

      2. Claptrap314 Silver badge

        Re: Linux and more

        You're directionally correct, but very wide of the mark.

        If you're not testing 100% of functional logic, you are NOT testing the logic. In production code, no branch point should exist except that a test requires it.

        Comments are not executed. Therefore, they are documentation and subject to becoming a false source of truth. No comment should be needed to explain what is being done, unless it is along the lines of

        # See https://en.wikipedia.org/wiki/A*_search_algorithm

        def a_star(graph, start, end)

        or, as mentioned elsewhere # This is to comply with policy blah

        In fact, comments are code smells. If you feel the need to explain your code, the strongly preferred response is to simplify it until it does not need an explanation.

        1. chuBb.

          Re: Linux and more

          100% is the goal (and required for some of the core systems we produce, but you work up to them you dont start there), but pragmatism dictates some is better than none (nothing we do is safety critical lives or bank balances dont depend on it), especially when more time gets spent covering edge cases and guarding against misconfiguration in other modules/systems, when/if a bug is found in the wild it gets fixed and tests accrete to cover it as part of the fix process...

          As for your view on comments are a code smell, well i completely disagree i find them to be an invaluable education tool, and memory jogger, picking up a collegues code for first time or my own from a few years ago, i might know the broad strokes of what its supposed to do, but the details may not be obvious, sure i can read the code and know what it does (and often how it does too), but thats not the same as knowing why....

          Can guess you would hate one of my candidate tests which has 30 lines of (obfuscated) production code, and the task is to capture its functionality into plain English.

          1. Claptrap314 Silver badge

            Re: Linux and more

            Depends on 1) what you mean by obfuscated and 2) what the quality of your production code is.

            But to the deeper points:

            Do you agree that comments are documentation?

            Do you agree with the aphorism "documentation is always out of date"?

            If you agree with these, what is your proposed solution?

            Mine is to simply the code until comments are not required to explain the "what"--unless you are implementing some complex algorithm. Comments about "why" can be informative, but, as out-of-date documentation, not to be trusted.

            Finally, my definition of "pragmatism" means leaving code that is no harder to maintain than it was to write. (Ideally, it should be easier.) That does require a discipline. It pays of very quickly.

    3. Tomato42
      Boffin

      Re: Linux and more

      hehe, you think that "Reviewing, documenting, or managing code isn't." isn't fun

      you didn't even mention "testing"

      1. Claptrap314 Silver badge

        Re: Linux and more

        That's because testing is part of writing.

        https://en.wikipedia.org/wiki/Test-driven_development

        1. Tomato42

          Re: Linux and more

          hahahahaha, ekhm, sorry

          so, where are those unit tests for the Linux kernel?

  6. John Smith 19 Gold badge
    Go

    Old rule of chess.....

    If you want to play better, play smarter opponents.

    Finding bugs in an expert devs code makes you pretty good.

    But if you don't you get to find out how a master coder thinks.

    I like to think of like an MMA sparing bout. A strong piece of code takes on all comers and is still standing.

    You want to play the game? You want a rep? Start studying the field.

  7. Robert Grant

    Yes, it is hard to get your email client to work

    Not-so-subtle swipe at that ridiculous thing going round about email being so hard it put off Linux kernel devs.

  8. IGotOut Silver badge

    No surprise.

    You don't get the"look I created this" satisfaction by reviewing.

    When was the last time the proofreader of a novel got recognised, or the sound engineer who removed that quiet little click in the background of a track that would have ruined it?

    Nope, no fame, but essential.

    1. Rosie Davies

      Re: No surprise.

      Speaking as someone who has been a sound engineer and reviewed a LOT of other people's code...it doesn't matter. There's the satisfaction of a job well done, whether that's listening to a piece of music you saved or seeing a piece of code romp through a scenario that would have choked it.

      Rosie

      1. Tomato42

        Re: No surprise.

        dabbling in both photography and photo restoration, I have to second the opinion: I get satisfaction from both making a nice photo and restoring an old, faded, one

        think of reviewing as teaching and you can get satisfaction from seeing other people learn

    2. Anonymous Coward
      Anonymous Coward

      Re: No surprise.

      Actually that's not the case.

      When a patch lands on one of the sub-system lists it isn't just accepted or dumped without feedback.

      The first version of series is usually unacceptable to someone and will get comments to that effect.

      The v2 of the series will usually state who suggested something was wrong and what was done to fix it so reviewers get credit there.

      Then when parts of the series is acceptable people, hopefully people with the standing to pull your patch in, will add their reviewed-by, acked-by and/or tested-by tags and those are in the mailing list archives with your patches and if you do another series to fix something else you will add the tags directly to your commits which makes them part of the git history.

      Then finally when you patch is merged into someone else's tree that person will fire a merge request to Linus in the next merge window and your commits will be recorded as being committed by that person.

      TL;DR; If you just wanted to get your name in the linux git history to impress people that don't know better you could actually do it without writing a single line of code. As far as I know you could even get your name in the MAINTAINERS file as a reviewer without submitting a single line of code.

    3. sw guy

      Re: No surprise.

      As a famous example, there is the A in RSA…

  9. Anonymous Coward
    Anonymous Coward

    A useful filter

    > was asked whether the reliance on plain-text email for submitting kernel patches for discussion was deterring new contributors.

    If someone can't handle an email based git workflow, I'm not sure they are at a stage in their development careers where they should be submitting Linux kernel patches unaided. Best thing to do in that case might be to get a mentor.

  10. Anonymous Coward
    Anonymous Coward

    Code Critic

    > It’s just like with music, you don’t start off writing music, you start off reading music and criticising music.

    Here is one for El Registro: start a weekly code review column. Just need to make sure you hire the right person for the job; that would be someone who could never cut it as a developer but knows the lingo and has a big ego (where do I send my CV?)

    1. Androgynous Cupboard Silver badge

      Re: Code Critic

      We all remember our favourite bugs. I remember spending hours working through a GDB stack trace on a friends university project that was, quite literally, impossible. Turns out he was overwriting the stack return pointer with something that was accidentally a valid pointed to elsewhere in the code. The look on his face when I figured that one out was wealth beyond measure (and yes, this was a long time ago),

  11. Unfrozen Cave Man Programmer
    WTF?

    So let me get this straight -- this person is suggesting that rank amateurs, who evidently can't find, install, or use a text-mode email app -- should be reviewing code? Kernel code? And forwarding what they think are good changes to more senior developers?

    Is she nuts?

    They'd be far better off training an algorithm to spot good revisions from bad ones based on thousands of submitted patches than leaving it to a million monkeys who can't send an email without Microsoft Outlook.

    Let the dilettantes scour the rejected patches for corn among the dung.

  12. Anonymous Coward
    Anonymous Coward

    "nobody wants to write an operating system"

    Actually, it's more "nobody wants to invest money to write an operating system", so why should someone do it when there's a free one?

    The Linux monoculture will become a huge liability in the future.

    1. Doctor Syntax Silver badge

      Re: "nobody wants to write an operating system"

      As opposed to the Windows monoculture being a liability right now?

  13. Anonymous Coward
    Anonymous Coward

    They should rewrite the kernel in rust.

    That'd be cool.

    To be honest, I have given zero thought as to who "they" would be, why "rust" (I was originally going to type "python") and how this would make anything better.

    But they should definitely do it.

    1. Anonymous Coward
      Anonymous Coward

      Re: They should rewrite the kernel in rust.

      It helps less experienced programmers produce less broken code.

      My vote would be for Elixir.

      1. _LC_
        Stop

        Re: They should rewrite the kernel in rust.

        Great, so all our software can be written by "less experienced programmers" then, as they are so much cheaper.

        Hint: In two years we will have Blahzoo. A completely new programming language, which avoids all the errors from previous languages...

        I'm too old. I've seen the repetition of this already.

        1. herman

          Re: They should rewrite the kernel in rust.

          Hmm, it is all ALGOL to me too...

        2. Anonymous Coward
          Anonymous Coward

          Re: They should rewrite the kernel in rust.

          Absolutely! All software in existence now is riddled with bugs. Anything that improves that situation can't come quickly enough.

          Humans really shouldn't be allowed anywhere near a computer.

          1. John Smith 19 Gold badge
            Unhappy

            "Humans really shouldn't be allowed anywhere near a computer."

            And with the endless levels of "managed" code very few are.

        3. EnviableOne

          Re: They should rewrite the kernel in rust.

          Which is why banking still runs on COBOL and kernels are written in C

    2. Anonymous Coward
      Anonymous Coward

      Re: They should rewrite the kernel in Lisp.

      Actually, that would let the kernel rewrite itself, so maybe not.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like