"Missed" traffic lights
Sometimes you will miss or otherwise not see a signal, due to sun blindness, something being in the way, or driven snow covering the signal (the downside of the new LED traffic lights is that they don't output enough heat to continually melt snow impacting them like the old school bulbs did)
But people are smart enough to take other cues, such as "if cars traveling the opposite way are stopped, I probably need to stop also" or "if cars are crossing in front of me, that probably means they have green". It is easy to train cars for the latter (they hopefully wouldn't cross a green light no matter what if there were cars traveling in front of them) but I'm not so sure about the former.
In intersections with separate left turn lanes, you might be green in your direction and the other direction will be red until yours turns red then they get green. Or they will be red for a time until all your left turners have turned. The car would need to be able to tell the difference between cars waiting to turn left because they either have a red light, or they have a green but must yield. Or there could be cars waiting to turn right, but they can't because cars traveling in your direction are turning left and have the right of way.
There are a lot of complexities that software can't easily capture. The worn lane markings will make that even more difficult, and that's not something easily solved. Where I live it snows in the winter, and the plowing and sand spread and so forth wear down the paint. They re-paint the lanes every year on major roads, but they get worn down to the point of almost disappearing in the next year. In the winter when there's snow on the roads you determine where the lanes are based on where cars have driven. If it is fresh snow, you are basically blazing your own trail and hopefully get it right so it doesn't confuse cars who follow you later.
I recall this past winter in such a situation I was driving in several inches of fresh snow on a street that is two way that becomes one way further on. I shifted over into the left lane as I needed to turn shortly after the one way transition but I guess I wasn't paying attention because I had shifted a block too early, and had to switch back because the little 'island' that directs traffic from another one way traveling the opposite direction to become the opposing lane of the two way was right in my path. If a self driving car decided following my tire tracks was a good way to know where the lanes are, it would risk driving into oncoming traffic if a car happened to be going in the opposite direction at that time. At the very least it would concern its passengers!