Yes, by definition. I did not invite them nor did anyone else. Therefore they impose themseves upon all of us. Unless the property is private it is impossible to know who is invited and who isn't. The people native to the place having others imposed upon them have the grievance. That thought experiment logic you wrote is akin to If you don't want a burglar in your house and he does want you to attack him who is at fault? Nonsense. Illegal immigrants are at the very least uninvited by the warlords who impose their will upon the people. As long as there are public borders the anarchist argument is kind of moot though.