Re: Clear cut...
"Even with a backup driver, the AI really needs to be very, very, good at avoiding accidents. This is going to require some rethinking of test protocols, even if AI surpass regular human drivers in safety"
Many of us have been expecting something like this to happen sooner or later if automated vehicles were allowed to be programmed by american drivers, due to the uniquely pedestrian-hostile environments and laws in that part of the world.
Humans don't like running things over (even animals), but if you program your vehicle with an assumption that legal road rules are really the way things are then you'll get robotic killing machines if people don't do as 'expected' and only cross at "crosswalks", or with lights.
Programmers from other countries know that pedestrians have priority on the road at all times and are legally allowed to step onto the road from anywhere, so will make sure the machines are setup to react accordingly - and ensure that if a pedestrian 40 metres ahead looks to be about to step onto the road, that the vehicle is already slowing down.
Arizona is one of those states which is one of the shittier places for pedestrian safety, which makes it a lousy place for testing robot cars - sure you can test how they go when things are 'normal', but you have very few exception conditions appearing to exercise the "non-normal" testing space and that means the supervising hoomun gets complacent, sleepy and too slow to react when things go pear-shaped.