Yes this has to be designed better to be intuitive with the privacy aspects. If you start a chat with someone, nobody knows about it. But adding someone to your contact list is the same as your follow list in other Nostr clients, so those are inherently public.