back to article Auto-makers told their autopilots need better safeguards

America's National Transport Safety Bureau (NTSB) has decided that late Tesla-driver Joshua Brown was responsible for the crash he died in, but that Tesla's Autopilot contributed by (at the time) allowing him to ignore the road for too long. As we reported in June, Brown's hands-off approach to driving was the big factor in …

  1. Brenda McViking

    It's a problem - Is the human in control, or is the machine?

    In the aviation industry, experience would tell us that even with intensive, professional training on the human side, an semi-autonomous system failing over to a human will often produce a startle effect which the human cannot process in the given time (which is usually tens of seconds in the air because you have height on your side- cars are going to have mere seeconds, if that). There are elements of mitigation involving clear display of information to said human, forcing them to take-over every now and then to stay attentive, and training them to deal with such situations, but they don't prevent the issue completely, and it's probably not feasible to give this type of training to your average driver as they aren't going to want to pay for it.

    Creating a safe semi-autonomous system is probably a lot harder than creating a safe fully-autonomous system, and until that changes, I think we'll be seeing more accident reports stating that the computer needs more control, humans need to RTFM, and wherever humans get involved, lots of holes appear in your layers of swiss cheese because they never behave as expected. I note however, that fundamentally, this accident was caused by the errors of 2 humans driving the truck and the tesla respectively - it's not going to sound a death-knell to autonomous driving aids.

    1. Voland's right hand Silver badge

      It is easier to automate the damn highway

      Frankly, it is easier to automate the highways, then automate driving in general.

      Here is an autopilot lane(s), please surrender your control to the highway computer and tell it which junction do you want to leave at. Next step - interconnect the systems. Sure, it will not be door-to-door, but it will be good enough to automate 95% of the journey.

      The issue is that no country is willing to even consider investing into something along these lines at this point.

      1. Lee D Silver badge

        Re: It is easier to automate the damn highway

        We had one. It's called the train network.

        Automated vehicles already operate all over parts of London (Docklands Light Railway for instance).

        What we need is a massive shakeup where we basically convert roads to railways (not literally, but the same kind of idea of join the network and leave and little opportunity to go wrong while you're on it) and give people automated cars (the word 'cars' works for both scenarios) that *can* use them or not.

        Massive infrastructure project, yes, but really it would put car companies out of business because nobody's car would be any different to the one next to it. Can't see it happening for political reasons, nothing to do with the infrastructure. Hell, how many billions were spent drilling a hole for HS2? You could probably repave a lane of every motorway for that price.

        1. Adam 52 Silver badge

          Re: It is easier to automate the damn highway

          We have 2,000 miles of motorway and it costs £10m/mile to add a lane. So that's £20bn. HS2 is £55bn and the tunnel part of that is £60m/km for 40km, so £2.4bn.

        2. Chemist

          Re: It is easier to automate the damn highway

          "Hell, how many billions were spent drilling a hole for HS2?"

          Do you really mean HS2 ? AFAIK no real work has been done at all yet.

        3. Stoneshop

          Re: It is easier to automate the damn highway

          Massive infrastructure project, yes, but really it would put car companies out of business because nobody's car would be any different to the one next to it.

          Er, no. People still have different transport needs: different maximum number of occupants, different load capacity, different maximum range, and people also have different spending room towards their vehicle(s). Plus, your automated highway won't control you all the way from door to door. Maybe that that last mile can be autonomously navigated by the car, but that depends on the circumstances.

          As long as cars have a standardised way of 'hooking in/out' of the 'train', it doesn't matter whether you're driving a 2CV, a Lamborghini or a Ford Transit.

      2. David Roberts
        Holmes

        Re: It is easier to automate the damn highway

        I have commented on this in the past but I'm just too idle to look it up.

        The big problem is the existing infrastructure especially in the UK and the suitability for upgrading to support automation.

        (1) Assume you will generally need at least two automated lanes in each direction, one for trucks with a limited maximum speed and the other for faster, lighter vehicles. Much like current dual carriageways.

        (2) Assume that you will need another two lanes to provide for the non-automated traffic during the (protracted) change over phase again to allow some segregation between cars and trucks.

        This gives you a minimum of a 4 lane highway in each direction to reap benefits from automation.

        Time, cost and complexity; you could look at the work to increase capacity on the A14 between the M11 and the A1M from the current inadequate two lanes each way.

        Real need for extra lanes; look at places for example on the M1, M4, and M25 where there are 4 lanes currently due to volume of traffic. Then add 2 more as a minimum.

        For uptake of new privately funded infrastructure look perhaps at the M6 toll north of Birmingham.

        As has already been suggested the requirement for an automated transport network starts to look very similar to a rail network with stations where you switch to an alternative infrastructure.

        So perhaps we should be looking at building new infrastructure akin to a rail network which only intersects occasionally with the current road network and easing the load on short journeys by enabling long distance automated transport. This would, of course, have to be significantly cheaper than just upgrading the current rail network.

        However in a country like the UK with such a large amount of established infrastructure and such a dense population this seems to be a logistical nightmare. Just bypassing a town or village on a single carriageway road seems to take decades to organise and upgrading single carriageway to dual carriageway on major trunk roads goes at a snail's pace.

        Countries with wide open spaces and massive wide roads (such as quite a lot of the USA) could be suitable for this next stage of transport evolution to make long journeys safer and more comfortable. I remain to be convinced that it is viable for most of the UK.

        1. Filippo Silver badge

          Re: It is easier to automate the damn highway

          Just a quick note, I'd be fine with having a single automated lane per direction, implying that everyone goes at lorry speed, if I'm allowed to not look at the road at all. I'd rather spend three hours doing something productive or sleeping, than two hours driving.

          1. Voland's right hand Silver badge

            Re: It is easier to automate the damn highway

            Just a quick note, I'd be fine with having a single automated lane per direction

            Bingo - start with the trucks too. The numbers are less and the economic and pollution benefits are staggering. There will be plenty of people willing to join in a car too despite it being 10mph headline speed slower.

            In addition to that a lane under computer control can be narrower too with significantly lower safety margins under bridges, in tunnels and on overpasses, so the cost of adding it is lower than the cost of adding a normal lane.

            1. John Brown (no body) Silver badge

              Re: It is easier to automate the damn highway

              "Bingo - start with the trucks too. The numbers are less and the economic and pollution benefits are staggering. There will be plenty of people willing to join in a car too despite it being 10mph headline speed slower."

              And if the lorries are under computer control, they'll all be going at the same speed with much shorter gaps between then and no 5 mile overtakes at 0.5mph speed difference so much more space available in the remaining lanes. And why should the lorry speed limit remain at 60mph on an automated motorway lane or artificially limited to 56mph if it's under computer control?

              It could be interesting though. If the automated lane is the left side, how do we make it easy for everyone else to exit safely at a junction? If the automated lane is the outside lane, how do we make it safe for the lorries to exit across 2 or three lanes of possibly faster moving cars?

              As Tesla and this report have discovered, and as many of us here pointed out after the initial reports of the accident, partial vehicle automation is harder for humans to cope with than full or no automation.

          2. Medical Cynic

            Re: It is easier to automate the damn highway

            Perhaps trucks could be allowed to drive faster, under automatic control in a designated lane?

      3. DanceMan

        Re: It is easier to automate the damn highway

        If only an equivalent amount of the effort currently being put into developing automated cars were put into automating traffic signals in urban areas we could have improved traffic flow (for buses as well as cars and trucks) and less fossil fuel usage and pollution. We have sensors imbedded in pavement to trip lights, but for a fixed interval, unable to sense when the cross traffic has cleared and lights can change again. Same problem for pedestrian lights.

        I agree that the ultimate solution will be an integration of both the vehicles and the roads.

  2. Snowy Silver badge
    Facepalm

    Still say

    [quote]While still evaluating the NTSB report, the electric car-maker added that “We will also continue to be extremely clear with current and potential customers that Autopilot is not a fully self-driving technology and drivers need to remain attentive at all times”. ®[/quote]

    Maybe they could start by not calling it an Autopilot and call it something that would make it clear it is something to help you drive not do it for you while you ignore the road!

    1. Adam 52 Silver badge

      Re: Still say

      You mean like Autopilot? So many people with no idea of what an autopilot does ready to criticise.

      1. RPF

        Re: Still say

        Some of us have plenty of autopilot time, thanks.

        20,000 hours enough?

        1. Adam 52 Silver badge

          Re: Still say

          Not quite sure the point you're making. If you really do have 20,000 hours and that's as a professional, then you really should be aware of the large number of incidents arising from aircrew failing to monitor the autopilot correctly - approximately 3 notifiable incidents per year and hundreds of dead passengers, notably Air France 447 and Continental 3407. If you think autopilot doesn't need monitoring (in an aviation, maritime or automotive context) then you are a danger to all those around you.

          But then anyone with a genuine 20,000 hours would have done crew management training on why relying on experience and seniority is dangerous too.

          1. RPF

            Re: Still say

            Yes I really do have 20,000 hours and was merely replying to your comment: "So many people with no idea of what an autopilot does ready to criticise."

            But go ahead and carry on arguing with yourself, because nowhere did I say that autopilots don't need monitoring.

            With a genuine 20,000 hours I can tell you that "relying on experience..." is absolutely NOT dangerous

            at all; it's the exact opposite.

      2. John Brown (no body) Silver badge

        Re: Still say

        "You mean like Autopilot? So many people with no idea of what an autopilot does ready to criticise."

        No, the problem with the term "autopilot" is the vast majority of the population don't know what it does and make assumptions that it's more capable than it is because their only experince of the term is from the media, mainly fictional TV. Just because you are a professional pilot with many years of experience and know precisely what an aircraft autopilot can do doesn't mean that the general population also know this.

        Most of us here know what computers and their software can do, but that doesn't stop the general population thinking you can zoom into a CCTV image and get a car registration number from a reflection in a shop window or someones eyeball or a single "enhanced" pixel.

    2. Anonymous Coward
      Anonymous Coward

      Re: Still say

      Maybe they could start by not calling it an Autopilot and call it something that would make it clear it is something to help you drive not do it for you while you ignore the road!

      I think other brands go with words like "road assist" and other phrasings that seek to avoid the "auto" part. In my opinion, the whole notion of going to watch a movie while you're on the highway with technology that has had less than a decade of joint use was at best a questionable lapse of judgement by the driver.

  3. Lee D Silver badge

    You can't get away with just a "don't use this product in that way" if it's blindingly obvious that people are going to use that product in that way, deliberately or not, if you don't protect against it and the consequences could be fatal.

    Seems quite sensible. You can't just make an open-bladed huge great cutting machine and then put a warning in the manual not to go near it. You have to have safeties and lock-outs and mechanisms and barriers. Why a tonne of machine moving at 100mph should escape such basic safeties was always beyond me anyway.

    And it was quite obvious that it was going to happen sooner or later and, now that you have to keep your hands on the wheel, these things are slightly safer.

    But, to be honest, I've seen drivers not pay attention to the road from the same lengths of time when they are driving a non-automated vehicle. That's how a lot of those "lorry hits line of stationary traffic" accidents occur, people literally not watching for long periods of time.

    To be honest, Tesla were always playing fast and loose with safety, and now it's starting to come back to bite them. Fortunately, in this case, the only death was the guy not paying attention. Are we really going to have to wait for some minibus full of schoolkids to die before we realise that other things they do are also dangerous and cause lax driving.

    1. Mongrel

      "But, to be honest, I've seen drivers not pay attention to the road from the same lengths of time when they are driving a non-automated vehicle. "

      Welcome to the M25

    2. Alister

      You can't just make an open-bladed huge great cutting machine and then put a warning in the manual not to go near it. You have to have safeties and lock-outs and mechanisms and barriers.

      And therein lies the problem, individuals no longer expect to have to take responsibility for their own safety, instead they expect that they will somehow be physically prevented from attempting something dumb.

      A prime example is railway crossings in the UK: despite barriers, warning lights, and all the other safety equipment, people still attempt to cross in front of a train, and the railway is somehow blamed if they are killed or injured.

      At some point people should have to face the consequences of their actions, up to and including Darwinism, and not expect that they should have been prevented from getting into that situation.

      1. rh587

        "And therein lies the problem, individuals no longer expect to have to take responsibility for their own safety, instead they expect that they will somehow be physically prevented from attempting something dumb.

        A prime example is railway crossings in the UK: despite barriers, warning lights, and all the other safety equipment, people still attempt to cross in front of a train, and the railway is somehow blamed if they are killed or injured."

        A very fair point.

        I recall a documentary on the railways and in one segment they were in the cab with a featured driver. He'd been forced to reduce speed so that his stopping distance was within Visual Range because there were reports of people fooling around on the line.

        He bemoaned the fact that this caused service delays. In the US, if you play on the railway and get hit by a train then the Police might - at most - collect your scattered body parts for your family to bury, taking the pragmatic view that you shouldn't have been there in the first place.

        UK? Train stops, Police examine the scene, passengers are parked for hours, following services are delayed.

        And the irony - in the UK we go to enormous lengths to keep people off the railways - fences, cameras, signage. Anyone on the railway is deliberately trespassing. There is no middle ground. In the US, they just run through towns and it's actually possible to wander onto a railway almost accidentally.

        1. lostsomehwere

          In the UK it's Network Rail's responsibility in law to make lines and access safe: if someone walks on to a crossing with their headphones on and doesn't look it's not their problem.

  4. Doctor Syntax Silver badge

    “conditions for which they were designed”

    Test tracks?

  5. Baldrickk
    Paris Hilton

    We can continue to make cars more idiot-proof

    but the universe will keep producing better idiots.

    1. Adam 52 Silver badge

      Re: We can continue to make cars more idiot-proof

      It will, unless we stop intervening to prevent natural selection removing the idiots.

  6. Anonymous Coward
    Anonymous Coward

    Tesla Autopilot drove straight into the side of a truck...

    It didn't even see the truck.

    It apparently didn't even notice the crash!!

    As evidenced by its lack of reaction and subsequent tree impact.

    From a tech news perspective, file under FAIL.

    Cross-file under HUBRIS.

    There's far more interesting Human Factors worth examining in the Engineering and Marketing decision making at Tesla HQ, than what happened on the day of this incident. Be a shame to ignore the Teaching Moment.

    1. Stoneshop

      Re: Tesla Autopilot drove straight into the side of a truck...

      It didn't even see the truck.

      That seems to be correct, yes.

      It apparently didn't even notice the crash!!

      As evidenced by its lack of reaction and subsequent tree impact.

      If you had read the NTSB report, which contains the car controller logs, it is quite clear that the car DID sense the actual crash. From 13:36:12.7 (US Pacific time, so 17:36:12.7 local time) through 13:36:25.8 the car's data logging reports "Vehicle alert consistent with collision damage", and a number of sensor fault/sensor missing messages, including "Brake controller CAN node is MIA". So even if a brake command was issued, there would have been no response.

      1. Anonymous Coward
        Anonymous Coward

        Re: Tesla Autopilot drove straight into the side of a truck...

        "Brake controller CAN node is MIA"

        Why does Tesla installed their brake controllers in the (missing) roof? ;-)

        More seriously, they really need to look at the damage tolerance of their designs. (In addition to fixing their hubris-laced Autopilot.) CAN buses run all over, perhaps even into the roof. Perhaps they should have thought about that, and installed a back-up trigger circuit for the brakes.

        Teaching Moment, not to be wasted. If they dare to insist that they're without (significant) fault in this tragic incident, then they're dangerously stupid.

  7. Harry Stottle

    Attention "heartbeat" required...

    in some of the software I develop I use deliberate random errors in certain dialogues, to spot humans trying to answer questions without inspection or thought.

    It occurs to me that something similar is required for the "Level 3" driverless cars (which are supposed to be able to handle almost all situations but still need close human monitoring). i.e. the software should regularly (but randomly) send false alarms to the control panel and measure the time and accuracy with which the human deals with them. If their response time exceeds a safe threshold, take the earliest opportunity to park the car and cede full control to the human (with an auto reset of, say, the next day?)

    1. Anonymous Coward
      Anonymous Coward

      Re: Attention "heartbeat" required...

      "...'Level 3' driverless cars (which are supposed to be able to handle almost all situations but still need close human monitoring)."

      File under: Inherently-flawed concept.

  8. Nimby
    Devil

    Call me crazy, but...

    I remember the days when cars might spontaneously combust. You're just tooling along when whoosh! Time to pull over and put the fire out under the bonnet again.

    Now we need to talk about secondary CAN bus backups in case a significant chunk of a vehicle is physically sheered off during a collision ... caused by TWO drivers who were not paying attention.

    When did it become a requirement for cars to be made so darn smart to protect the people who became so damn stupid?

    I'm thinking the only problem with the car was the nut loose behind the steering wheel.

    Driving is always a potentially lethal endeavor. Anyone who forgets that driving is dangerous by watching an in flight movie instead of watching the road deserves everything that happens to them when a lorry cuts them off. Better them than the six year old kid running out into the street to get their ball. Driving is a privilege, not a right.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like