So I have a plugin for only one specific agent. I do not want that other people use it, only the people who are working with that agent.

I also don't want to take away all plugins from them to use for their chats. People can still choose to use Dall-E for example.