Fair enough. A $5 wrench is one of the most accessible "hacking" tools around, and a lot of people fail to consider it:
https://xkcd.com/538/
To be fair to Yubikey, you can have multiple keys and "cold wallet like" backups, as well as good rotation procedures if you lose one. Plus you can setup things so that sessions last longer and you don't need to carry your keu with you (at the cost of security, of course). This still won’t protect you from a $5 wrench, but even though I’m a big guy, I don’t think keeping a password stored in my head offers much additional protection if things really get rough...
On the convenience vs security scale, maybe what works best for you is a password that’s easy to glide type:
https://xkcd.com/936/
And since this post includes not one but two XKCD references, you know it’s solid advice 🤣🤣.