Setting for Max Tokens Output to be the Maximum Allowed by the Model
S
Scott
The models are beginning to diverge significantly in terms of how many output tokens they will provide. I would like the ability to easily set the max output tokens to the maximum for each model.
I can currently set the max number of tokens for ALL models (the Max Tokens setting in the Manage Models settings) or the max tokens for EACH AI agent. The problem with this is that I do things where I want the max tokens to be set to the maximum allowed by the model. It is very time consuming to switch from one model to another and increase/decrease the max tokens in the applicable settings accordingly.
It would be great if the "Max Tokens" setting for all models (the setting on the Manage Models settings) AND for each AI agent had a setting that would allow the maximum output tokens supported by the chosen model. This way, I can simply change the selected model I'm using (or the AI agent is using) and know that the output I get can be as much as allowed by the model without going into the settings and manually changing it each time.
C
Charcoal Locomotive
+1
N
Nigel Powell
agree
Diego Sala
+1!!!
It seems to me that leaving Max Tokens to "Default" limits (cuts) the output in some models like Claude 3.5 Sonnet, that I use the most.
I often want to set every model to its maximum number of max tokens, and as I often use to regenerate with different models, in order to set them to their limit of max tokens I have to change every one manually each time.
What about a new "maximum" setting for "max tokens", in addition to Default (blank)?
Note: as far as I understand it, the only drawback of setting it to its max output limit is that it reduces the max input length of the same amount of tokens. However, with the large context window of recent models, this is rarely an issue.
I also heard that it reduces speed and/or latency, but I evaluate the difference as inexistent or negligible.
D
Derek Breese
I echo your sentiments. Autosetting the max tokens based on the model is definitely needed,
R
Robert coorey
Agreed!