Wondering if that would leak private key material like nip-04 did, also would people be signing giftwrap with the conversation key than with an ephemeral key?

Wrong, the only scalable and decent way to prevent against spam is by picking relays that enforce limits and reading from such relays on a case-by-case basis. If you rely on an "authors" filter and blind relays you're limiting yourself to only ever seeing stuff from people you know and never learning that other people exist in the world -- or every client has to perform complex and costly computations locally that then make that "authors" list so big that relays will rightfully reject it. In other words: enforcing restrictions once per write is much more effective and efficient than trying to enforce them on every read.