It's a kind of protocol that exists within a more overarching protocol. For instance, if English is a protocol for communication, then standards for professional writing are a subprotocol within English. Similarly, Nostr is the overarching protocol and how to use relays and specialized notes for creating groups is a subprotocol within Nostr.

Loosely a protocol inside another protocol.

Nostr transmitted by Nostr; Nostrception