Re: Having met two of them...
For a long time there seemed to be a media consensus that Peter Davison's portrayal was a disappointing, lightweight followup to Tom Baker that started the show's slow decline in the 80s. It's interesting that this seems to have changed in the past few years, with far more people taking a positive view.
It's probably fair to say that whatever one thinks of Baker, the length of his reign and the strong character of his portrayal defined the Doctor for large swathes of people- *anyone* following him was going to have a hard time. I think the change is down to those of us not old enough to remember Baker well- if at all (and thus not having judged Davison in his shadow to the same extent) now being established adults, as well as many of those who watched the shows later being in a similar position.
I started watching Doctor Who at the tail end of Tom Baker's reign. (#) But- sacriligeous as it is to say this (##)- I preferred Peter Davison and he's probably "my" Doctor. In some respects I can understand why those who grew up with Baker felt that way- but Davison was an intentionally different Doctor.
Colin Baker I didn't like at all in the beginning. IIRC that improved slightly, to the point I was slightly sorry to see him go. I've seen bits of his stories since, and I'm still not sure to be honest, but it's hard to judge him given the ultimately misjudged character development he was given, along with the then-state of the show in general. In hindsight, it's clear some of the problems I associated with the McCoy era (pantomime tackiness) started here.
Sylvester McCoy- maybe lacked a bit of gravitas for my taste, but again, hard to judge fairly- the majority of the dislike aimed at him probably had more to do with the scripts and the state of the show itself. I remember being unhappy about the increasingly pantomime-like direction it was taking during his era, but in hindsight it had actually started to improve in its final season. (###)
(#) I have vague memories of the "new" starry title sequence replacing the "classic" tunnel one just before I turned five, which implies I- barely- remember the former too, but Baker's final "starry" season is the only one I really remember. IIRC I think I first watched it for the Daleks.
(###) I remember thinking Survival was the best story in a long time, though I didn't realise it was the final one until years later, as the show hadn't been publicly cancelled at that point.