Great question!

I look at this sort of thing based on the stability of products formed - technically a carbonyl has some nucleophilic character at its oxygen lone pairs, but if you think about the product formed with that attack, you'll have a formal positive on an oxygen which cannot get dissipated by deprotonation, because there are no protons to pluck.

With an amine or an alcohol, you will have that same transient formal positive after nucleophilic attack, but you can deprotonate to stabilize the product formed. You don't have that ability for a carbonyl, so it doesn't act as a nucleophile (it'll tend to leave right after it attacks if it ever does).

Hopefully that isn't too confusing, please let me know if a picture would help.