refusal field in response breaks response

[mase-ppi]

Bug description
All calls to OpenAIs chat completion since some hours seem to fail, since the reply from openai now includes the refusal field. In normal situations this field is returned "null", which is a bug from openai or a new feature, i guess? So, for me, nothing is working anymore.

Environment
Spring AI version 1.0.0-SNAPSHOT
Java version 17
OpenAI tested with function calls and gpt-4o and -mini

Steps to reproduce
I assume, just test against current chat completion api

Expected behavior
field is recognized

[mase-ppi]

Exception looks like:

org.springframework.web.client.RestClientException: Error while extracting response for type [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion] and content type [application/json]
at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:236) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.readBody(DefaultRestClient.java:667) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntityInternal(DefaultRestClient.java:637) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntity(DefaultRestClient.java:626) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.ai.openai.api.OpenAiApi.chatCompletionEntity(OpenAiApi.java:920) ~[spring-ai-openai-1.0.0-20240805.173614-443.jar:1.0.0-SNAPSHOT]
at org.springframework.ai.openai.OpenAiChatModel.lambda$call$1(OpenAiChatModel.java:224) ~[spring-ai-openai-1.0.0-20240805.173614-443.jar:1.0.0-SNAPSHOT]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:335) ~[spring-retry-2.0.5.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211) ~[spring-retry-2.0.5.jar:na]
at org.springframework.ai.openai.OpenAiChatModel.lambda$call$3(OpenAiChatModel.java:224) ~[spring-ai-openai-1.0.0-20240805.173614-443.jar:1.0.0-SNAPSHOT]
at io.micrometer.observation.Observation.observe(Observation.java:565) ~[micrometer-observation-1.12.4.jar:1.12.4]
at org.springframework.ai.openai.OpenAiChatModel.call(OpenAiChatModel.java:221) ~[spring-ai-openai-1.0.0-20240805.173614-443.jar:1.0.0-SNAPSHOT]
.....
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:925) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:830) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.5.jar:6.1.5]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) ~[spring-webmvc-6.1.5.jar:6.1.5]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[tomcat-embed-core-10.1.19.jar:6.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.5.jar:6.1.5]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.19.jar:6.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.5.jar:6.1.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.5.jar:6.1.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.5.jar:6.1.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unrecognized field "refusal" (class org.springframework.ai.openai.api.OpenAiApi$ChatCompletionMessage), not marked as ignorable
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354) ~[spring-web-6.1.5.jar:6.1.5]
at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213) ~[spring-web-6.1.5.jar:6.1.5]
... 64 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "refusal" (class org.springframework.ai.openai.api.OpenAiApi$ChatCompletionMessage), not marked as ignorable (5 known properties: "content", "name", "tool_call_id", "role", "tool_calls"])
at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 13, column: 8] (through reference chain: org.springframework.ai.openai.api.OpenAiApi$ChatCompletion["choices"]->java.util.ArrayList[0]->org.springframework.ai.openai.api.OpenAiApi$ChatCompletion$Choice["message"]->org.springframework.ai.openai.api.OpenAiApi$ChatCompletionMessage["refusal"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1138) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2224) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1719) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1669) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:546) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1419) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1419) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1419) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481) ~[jackson-databind-2.15.4.jar:2.15.4]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395) ~[spring-web-6.1.5.jar:6.1.5]
... 66 common frames omitted

[TarasVovk669]

I have the same issue

[cocomongg]

me too. I have the same issue.

[tzolov]

@cocomongg , @TarasVovk669 I'm unable to reproduce this issue nor our OpenAI integration tests.

The OpenAI API reference documentation does not mention "refusal" field either.
So i'm reluctant to introduce new non documented field.
Would the @JsonIgnoreProperties(ignoreUnknown = true) annotation on the ChatCompletionMessage record resolve the error for you?

[tzolov]

Can you please add the spring.jackson.deserialization.fail_on_unknown_properties=false property to your application.properties file and let me know if it resolves the problem?

[mase-ppi]

@tzolov did you test against an own gpt-deployment or official openai hosted? It also needs to be 4o or 4o-mini.

i think it is something like: https://platform.openai.com/docs/guides/structured-outputs/refusals

but in our case, refusal ist just delivered as null.

But yes, would make sense to ignore unknown fields in order to be more rebust against new changes.

Thanks

[tzolov]

@mathekcbo, thank you for the reference. I'll have a look

[mase-ppi]

@tzolov the property didnt work for me - just to clarify, I tried:

spring:
  jackson:
    deserialization:
      fail_on_unknown_properties: false

[tzolov]

@mathekcbo , thank you for testing.

Regarding:

@tzolov did you test against an own gpt-deployment or official openai hosted? It also needs to be 4o or 4o-mini.

Our ITs are running against the official OpenAi hosted and they include 4o and 4o-mini.

I'm puzzled why I'm unable to reproduce it. What user question would cause the "refusal" field to be set?

[mase-ppi]

@mathekcbo , thank you for testing.

Regarding:

@tzolov did you test against an own gpt-deployment or official openai hosted? It also needs to be 4o or 4o-mini.

Our ITs are running against the official OpenAi hosted and they include 4o and 4o-mini.

I'm puzzled why I'm unable to reproduce it. What user question would cause the "refusal" field to be set?

"How are you" (in my case: "Wie gehts dir?") - acutally i cannot see any prompt not returning it so far

[mase-ppi]

@tzolov any possibility to print out my stuff going to the openai api without debugging everything? Maybe the request/options are different, as only a few users seem to have this problem

[TarasVovk669]

I added an annotation to the model

[tzolov]

related to #1196