Ephemeral plugins
a
plugins should have an "ephemeral" toggle setting so that once its used in a response, it isn't continually sent in future interactions. things like google searches don't need to be carried around after the initial search basically. this saves a lot of costs.
speculation: perhaps the "unused" state would be indicated somehow rather than actually deleted, with an option to include it again
Nathan
Is this even possible? I thought all LLMs required the entire conversion to be resent for every message. Thats why per-message costs increase as a conversation goes on. I’ve had a $36 conversion before 😅
If this is fixable that would be huge.
a
Nathan its totally possible. you just.. refrain from sending some messages when you send it to the server for the next turn in chat. currently i am forcing this behavior (in an annoying fashion) by having an extension that automatically deletes plugins outputs in real time as soon as its done "using it"
Nathan
I meant without degrading the quality of the LLM‘s output.
There’s usually a reason why it is the way it is. These models are all about optimization.
Or do you not want it to have memory?
a
Nathan what you are proposing is known as a false dilemma. i think memory is great and it should have that. this suggestion would actually increase your usable memory. it won't degrade the quality of the LLM's output either.
heres a use case:
User: whats the name of the guy taylor swift is dating?
Bot: <calls search and other plugins many times, each with tons of text, as it researches and tried to find the answer>
Bot: she is dating xyz, who is a football player.
We no longer need that "research" history since we already got the answer. all those plugin data would only make every subsequent interaction cost you more money.
Nathan
a I see, you don't want it to remember. I use ChatGPT for such things.
Thinking out loud: How would the bot know whether or not to keep memory? He'd have to make a "one shot mode" you can enable that clears the context after every entry. Then it would only be enabled for that chat, which is reasonable if you're doing research for an extended period.
I can see this being a helpful feature.
Until he implements this, do you know about the "Clear Context" button? I think this is the manual way to do it. If you don't want to use your mouse, you can make a macro that clicks it. Just a thought.
a
Nathan i think you're catching on. basically we just don't want it to remember the scratch work that won't be useful after the first time. clear context wipes everything.
Nathan
a No need to be patronizing. You asked for a "per plugin" setting which made no sense. It actually needs to be a "per conversation" setting.
Anyway, all good 👍🏼
a
Nathan i think we are on different pages. the desire is for it to be per plugin over per conversation (though that could offer different value potentially)
B
B T
Good idea.