Repository navigation
Expose the model's reasoning from ChatCompletions API #625
Description
Activity
Thanks for looking at this. I am still finding this problem a point of friction when using OpenAI's models or VLLM models through this SDK. It would be handy to have this 😄
A current-main workaround for provider-specific streamed delta fields is
chunk.Choices[i].Delta.JSON.ExtraFields, without editing generated SDK structs. I verified the actualChat.Completions.NewStreamingpath onea1f7a2e1c56feb655eb8dc6a8ad45e57d3e93a8, Go 1.27.1 darwin/arm64, using controlled in-memory SSE fixtures.Four cases passed with one decoded chunk each and no stream error:
reasoningcontaining a fictional Unicode string: exact raw value retained and decodable as a string.reasoning_contentcontaining a fictional alternate string: exact raw value retained under that separate key.reasoning:null: field present with raw null.reasoning:7: field present with raw 7; decoding it as a string correctly fails.
Ordinary delta.Content stayed intact in all four. Access the provider's actual field name rather than assuming reasoning and reasoning_content are interchangeable:
field, present := chunk.Choices[i].Delta.JSON.ExtraFields[providerField] if present && field.Raw() != "null" { var fragment string if err := json.Unmarshal([]byte(field.Raw()), &fragment); err != nil { // handle unexpected provider value type } else { // route the fragment according to your application's policy } }
Check the choices bounds before indexing. Go's JSON decoding of null into a string yields no error and an empty string, so the explicit null check matters if presence/null/empty string have different meaning in the application. This is per-chunk extraction; accumulation, choice indexes and display policy remain application decisions.
go test -mod=readonly -run TestDharmaReasoningExtras -v -count=1 -timeout=30s .Network was denied and all content was fictional. No model, vLLM server or live API was used. This verifies preservation of these unknown delta fields through current streaming decoding, not provider support, an OpenAI reasoning-field contract, generated typed-field acceptance, or how the SDK accumulator handles extras.
This is some low-hanging fruit that will improve the library's usability for interacting with vLLM models.
If we were to add a
Reasoningfield to theChatCompletionChunkChoiceDeltastruct definition, as shown below, then we would be able to access the returned reasoning, if any.I would find this useful since it would allow me to return thinking content to the caller so they can see the reasoning process and know that the model has not hung unexpectedly, and also monitor that it is answering the prompt as intended.