Is the antenna outside or at least near a window? Indoors the signal will be poor at best. If it's from seeedstudio they tend to have example code to check out. You'll get it.

None of those GPU through hole header pins look soldered?

GPIO*

Indoors. But the garbled text is related to some decoding issue, not the GPS itself. I'm following Waveshare's tutorial (as it is one of their devices)

You're correct they aren't but I'm sure they're making contact. It was a tight squeeze so there''s friction there. Maybe its lose somewhere, but feels very tight to me.

I hear you, but those through holes usually oxidize very quickly in my experience, i've gotten boards back from production and them already have such crappy flow job they have corrosion appearing. I would highly suggest you solder, at least the power pins up. It's hard to get enough current for GPS. I've also had experiences with GPS and radios on RPIs not delivering stable power over the GPIO pins they've needed more filtering capacitance.

So potentially I could have a flaky connection with this current setup?

Very much so yes!

I think this one was meant to be pushed o though with no soldering, like it is a tight fit and came with this mounting component

I can see the air gaps around the pins. I highly doubt it as that simply looks like a pre-tinned through-hole board, which is designed to have the header pins soldered.