Secure elements are really just secure PIN checkers that release decryption keys. When I was naive I thought the signing operations occur off the main MCU and in a secure enclave, but this is not the case with most hardware wallets. A successful PIN releases a decryption key which then passes onto the main chip to decrypt the secret key in memory.