I didn't realise it was only limited to attending their house.

Yeah. It doesn't say they can't be arrested or anything. Rather they cannot be detained to slow a vote. After the vote, they can absolutely be arrested however.