The point is the *reasonable expectation of trust. You trust elliptic curve multiplication only goes one way and cant be unwound. It is a *similar application of trust to use a RNG. Is it necessary? No, use dice if you wanna. But if HWRNG is compromised then the entire security layer of the internet is broken and we have much bigger problems.