How else does Cove know which balances to fetch?

Ah, I better understand now, your addresses are doxed unless you use a private node. Your IP is doxed no matter what for price discovery (which is reasonable).