THIS. You just nailed it.For some it's that fear of not trusting themselves with a vehicle that easily weighs more than 15-25 people and can cause serious harm if not handled with lots of care. One could potentially say it's a self-confidence issue.
Personally I've never liked driving because I am too aware of my own mind drifting pretty much everywhere.
I also have strong tendency for mind drifting. And I feel that I need to hold all the factors of driving in an immediate focus, all the time. If I let go, I'm afraid I'd lose control. Probably also because of this I could never get driving on "autopilot" in my mind.
Add to this a good degree of physical clumsiness... I learned to ride a bicycle a good couple of years after all other kids, and with a lot of effort and pain (literally, as I was falling of it many times until I learned).
I think I'll try to contact some driving school(s) about a "reminder" course, and have a test drive, in April maybe?... Yet another factor is that back then, I was coerced into getting the licence, so I had negative stance towards the whole thing. Nowadays there finally is some intrinsic motivation, even as it has to compete against anxiety. I need to try and see.