I think there was something going about last year where disabling internet connectivity checks stopped an IP leak but i'm not sure if this is related or I misunderstood a previous concern

This

They would have, but we have heavy disagreements with Mullvad on how they phrase this as they fixed it within their app. If it is possible to set up apps in a way that they don't leak without OS changes then it was an app issue, it's premature to blame the OS. They are being unclear about this. The Android OS VPN implementation is unaffected. The OS could also prevent these leaks but it is possible they may not had viewed this in scope of the feature. DNS is handled in a special way and the VPN gets to set DNS separately from the VPN and can send it through it or outside it, etc. VPN app developers should also be testing these basic cases themselves already ("only affect certain apps") and it appears they had not. As for the second case ("For a short period of time while a VPN app is re-configuring the tunnel or is being force-stopped/crashes"), this is being investigated. It sounds like an OS bug but a leak is not inherently responsible by the OS. Fortunately that second example is very limited. It is also worth noting they did not discover these issues first rather they were reported to us by a GrapheneOS user which we posted about days before them. We are also aware of a local network multicast leak which is an actual OS bug which they haven't mentioned. Also see:

It's worth noting that the built-in VPN support doesn't have these leaks. We don... | Hacker News

Mullvad are also linking an older article regarding a connectivity check "leak" which is misleading. That connectivity check is needed for determining which networks work, and triggering captive portals the user can interact with to log into WiFi networks with login pages. This would help you deal with the captive portal *without* disabling the VPN which would make everything else leak. GrapheneOS has also had the option to disable or change it for a long time.

This is different and was also misleading on Mullvad's part, it's intentionally designed that way. We make an explanation on that here:

Frequently Asked Questions | GrapheneOS

You can also turn that off in GrapheneOS.

Makes total sense. I also have disagreements with the way Mullvad handles a few things, so this kinda checks out. Thank you :)