Of course you can do all of that and then have a makeshift "probably secure" signer for not very critical stuff. But the sheer amount of components in a smartphone will always be a blind spot imo. Building your own dedicated signer from general purpose hardware used with peer reviewed, and proven open-source software will always be a superior approach, just less convenient.