Non-required serializer related fields

[andrewyoung1991]

Checklist

• I have verified that that issue exists against the master branch of Django REST framework.
• I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
• This is not a usage question. (Those should be directed to the discussion group instead.)
• This cannot be dealt with as a third party library. (We prefer new functionality to be in the form of third party libraries where possible.)
• I have reduced the issue to the simplest possible case.
• I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

Steps to reproduce

create a serializer that subclasses serializers.Serializer with a non-required primary key relation

class SearchSerializer(serializers.Serializer):
    q = serializers.CharField()
    user = serializers.PrimaryKeyRelatedField(queryset=User.objects.all(), required=False)

call the serializer in with a request that doesn't have a user parameter

serializer = SearchSerializer(data={"q": "hello world"})
serializer.is_valid(raise_exception=True)
print(serializer.data)  # <- KeyError 'user' is raised here

Expected behavior

the serializer should validate that the field q exists and ignore validating the user field if it is not supplied in the request data. The an example request

{
  "q": "hello world"
}

should be valid, and user should either be None or not included in the serializer.data

Actual behavior

the serializer raises a KeyError on serializer.data if user is not included in the request data.

[xordoquy]

From the documentation:

Relational fields are used to represent model relationships. They can be applied to ForeignKey, ManyToManyField and OneToOneField relationships, as well as to reverse relationships, and custom relationships such as GenericForeignKey.

As such, using PrimaryKeyRelatedField to create a relation between a ModelSerializer and a regular Serializer should not be used.

[nazarewk]

I encountered exactly the same issue, but managed to get it to fail with ModelSerializer.

Here is the whole PR along with the fix #3984

[lovelydinosaur]

The issue looks valid to me as stated.
Nothing wrong with using relational fields on regular serializer classes.
(If anything I'd far rather see bugs reduced to the plain Serializer case, rather than using ModelSerializer, which just masks what set of fields are being used.)

[lovelydinosaur]

Turns out this isn't valid.
required=False does work on a relational field.
What's going wrong here is that you're calling serializer.data, not serializer.validated_data.
required=False ensures that the field does not have to be present in order for the field to validate incoming data.
However does not indicate that a instance to be serializer can omit the value.

[yevhen-m]

@tomchristie actually it does say so.

[lovelydinosaur]

True.

[carltongibson]

I'm going to de-milestone for now. We can reassess after v3.7

[cgthayer]

:-O Wow,

As a new user, I found it very surprising that I needed to set an entry in data to None to get this working. Especially since the docs say "will not take place if the field is not INCLUDED" (http://www.django-rest-framework.org/api-guide/serializers/#validation)

Note: If your <field_name> is declared on your serializer with the parameter required=False then this validation step will not take place if the field is not included.

What would also help is a code sample around

• http://www.django-rest-framework.org/api-guide/fields/#required
• http://www.django-rest-framework.org/api-guide/relations/#primarykeyrelatedfield
• Somewhere in http://www.django-rest-framework.org/tutorial/1-serialization/
• http://www.django-rest-framework.org/api-guide/serializers/#modelserializer

However, the design has a problem: If you are adding fields over time, and you want to maintain some backward compatibility, then it should be safe to add them as require=False without updating all the code to include a new field with a None value.

class ContactsListSerializer(serializers.ModelSerializer):
    group = serializers.PrimaryKeyRelatedField(queryset=models.Group.objects.all(), required=False)

class ContactsListViewSetTests(ApiTest):
    view_class = views.ContactsListViewSet

    def test_contacts_list_create(self):
        view = self.view_class.as_view({'post': 'create'})
        data = {
            'display_name': 'Default List',
            'list_tag': 'default',
            'group': None,  # required to be in, but ok to be None                                                                                                                                                                                                                                                                                                                       
        }
        s = serializers.ContactsListSerializer(data=data)
        if s.is_valid():
            logger.info(f'xxx is valid {s.validated_data}')
        else:
            logger.info(f'xxx is not valid {s.validated_data} errors={s.errors}')

I'm at djangorestframework==3.7.7
Not sure why this didn't fix it: #3984

[lovelydinosaur]

it should be safe to add them as require=False without updating all the code to include a new field with a None value.

If you’re removing the relationship that the serializer field points at, then just remove the serializer field too. (Or at least switch it to read_only=True)

[cgthayer]

it should be safe to add them as require=False without updating all the code to include a new field with a None value.

If you’re removing the relationship that the serializer field points at, then just remove the serializer field too. (Or at least switch it to read_only=True)

I'm referring to the case of adding a field to a model. If one adds a field to a model, and a adds it to a serializer as required=False, my expectation is that users of the library would be Ok (backwards compatible). I believe that these clients break because the data dict passed to Serializer(data) needs the new key, otherwise validation fails. In other words, adding a new field to a model (and serializer) requires all clients be upgraded at the same time.

I'm thinking of how backwards compatibility is handled in networking code, such as Thrift and Protocall Buffers. But perhaps I'm misunderstanding and my code is just wrong?

[TauPan]

@tomchristie actually it does say so.

For reference, if other people have trouble with not required values not being omitted from the output,
I'm using this as a workaround (first naive implementation, I'm not looking at defaults or allowed_ here):

class OmitNoneNotRequiredSerializer(object):

    def to_representation(self, instance):
        ret = super().to_representation(instance)
        for field in self._readable_fields:
            if not field.required and ret[field.field_name] is None:
                del ret[field.field_name]
        return ret

Just add this as a mixin (first class argument) to your serializers to omit every None from the json output:

class FooSerializer(OmitNoneNotRequiredSerializer,
                    serializers.HyperlinkedModelSerializer):
...

But this raises the issue how the behaviour should be if allow_blank, allow_null etc. are set. Maybe it would be better to add a more explicit argument to the Field, such as e.g. allow_omission=True/False.