How to skip an intermediary model?

[carltongibson]

Hi Folks.

I've got three models Start, Middle, and End.

I'm serialising Start, and need to include End, but I can't quite work out how to skip Middle.

The relationship would be middle__end_set but I can't seem to write the right projector/pair for this. (The prepare bit is simple enough: `qs.select_related_fields("middle__end_set") but I'm to seeing how to write the projector.

I don't suppose you have an example of this do you?

Thanks 🙏

[carltongibson]

OK, so I've got this working but currently it's a little long-winded.

Here's a diff to an example from test_pairs.py that does the sort of thing I have in mind:

diff --git a/tests/test_pairs.py b/tests/test_pairs.py
index 0c9bbfb..5db53d2 100644
--- a/tests/test_pairs.py
+++ b/tests/test_pairs.py
@@ -537,10 +537,18 @@ class PairsTestCase(TestCase):
         )
 
     def test_select_related(self):
-        owner = Owner.objects.create(name="test owner")
+        group = Group.objects.create(name="test group")
+        owner = Owner.objects.create(name="test owner", group=group)
         Widget.objects.create(name="test widget", owner=owner)
 
         prepare, project = pairs.combine(
+            (
+                qs.pipe(
+                    qs.select_related("owner__group"),
+                    qs.include_fields("owner__group__name"),
+                ),
+                projectors.noop,
+            ),
             (
                 qs.pipe(
                     qs.select_related("owner"),
@@ -561,6 +569,21 @@ class PairsTestCase(TestCase):
                     ),
                 ),
             ),
+            (
+                qs.noop,
+                projectors.producer_to_projector(
+                    "owner_group",
+                    producers.relationship(
+                        "owner",
+                        producers.relationship(
+                            "group",
+                            projectors.producer_to_projector(
+                                "name", producers.attr("name")
+                            ),
+                        ),
+                    ),
+                ),
+            ),
         )
 
         with self.assertNumQueries(0):
@@ -573,7 +596,11 @@ class PairsTestCase(TestCase):
             result = project(instance)
 
         self.assertEqual(
-            result, {"name": "test widget", "owner": {"name": "test owner"}}
+            result, {
+                "name": "test widget",
+                "owner": {"name": "test owner"},
+                'owner_group': {'name': 'test group'},
+            }
         )

From the Widget we want to include a representation of the Group, but have that at the top level, under owner_group here, rather than nested under owner.

Having the nested producers.relationship() calls is a pain. It would be nice to be able to pass owner__group there. (But AttributeError: 'Widget' object has no attribute 'owner__group', of course.)

The pattern otherwise makes sense, but it takes some thinking about. (That might just be How it is™)

@j4mie, could I ask, do you have any suggestions of how I can do this better? 🎁

[j4mie]

Hi Carlton!

First, apologies for missing this originally. I got the notification when you tagged me, but not for the original discussion. Will check my settings.

I'll give this some proper thought later, but as an initial comment I thought it'd be useful to share an excerpt from our internal best practices document that seems relevant here. I'd love your take on it. Note this is mostly oriented around developers building backend endpoints to support frontend features in a React app, rather than building a public API - is that what you're trying to do?

Endpoint Data Shapes

When creating backend read (GET) endpoints to support frontend features, it’s tempting to design a bespoke data shape that is “helpful” to the frontend, by pulling up fields from nested related object(s) onto the main root object(s), aliasing fields to provide names that more closely match frontend concepts, and so on. At first glance, this seems to match the philosophy of “pushing complexity down”, i.e. have the backend handle the reshaping and provide the frontend with a simpler structure

However, there are a couple of reasons why this isn’t a good idea (which are explained below). Instead, you should strongly prefer to return the data in a shape that matches the underlying backend data model as closely as possible, with little or no aliasing of fields or reshaping of data from nested related objects.

The reasons are:

(1) Although it means your frontend code is more “complex” in the sense that it doesn’t receive the data in a format that is “simple” in terms of the developer writing that code today, overall it makes the system simpler to read, understand and modify. In other words, the total complexity across the frontend and backend is significantly reduced. This is because the developer reading the code later can expect that the data format will always match closely to the backend data model, so they only have that one model to hold in their head, not whatever various transformations were applied in various different parts of the code. Having tried both approaches in large, complex projects, the latter approach (avoiding data reshaping) led to much more maintainable and understandable code on both the frontend and the backend.

(2) django-readers is very much oriented around this idea. A django-readers spec that reflects the underlying data model tends to be simple and high-level, mostly expressing field names and relationships through the built-in shortcuts, rather than requiring lots of custom code. Custom reader pairs, when they are needed, tend to operate on individual model fields rather than relationships to other models, making them much simpler to write. Although django-readers does support reshaping data (it is flexible enough that you can do anything you need to, eg if you need to conform to existing frontend code or to replace an existing stable API), this tends to result in more custom code that is more difficult to understand, and results in a steeper learning curve for new developers. Extensively reshaping data is going “against the grain” of django-readers: it is much easier to use if you avoid doing so.

Of course there are exceptions to every rule. For example, if the backend needs to return values that are derived from the underlying relational model via a transformation, or if the frontend needs aggregated information (a count of the number of users inside each group, say) then it does make sense to return a bespoke field from a custom reader pair. But the underlying relational model itself should remain intact wherever possible.

[carltongibson]

Hey @j4mie — thanks for that excerpt. Seen 😅

I'll review the Why for needing this. Perhaps we can just drop it in the specific case. 🤔 (I'll follow up when I've had time to consider fully.)

I still be interested in your take on my example though. It seems like pulling up an attribute (or a producer value) should be doable.

I have half-formed visions of the Django-Readers approach being an answer to Django's long-outstanding serialisation Question — shaping the data seems important there. (I could also imagine an extract step followed by a reshaping step though, or something, so it's just ideas/exploratory there at the moment.)

[carltongibson]

I'll follow up when I've had time to consider fully.

OK, for the moment we're going to go with the more vanilla readers code. We'll do a bit of mapping in a simple Python object before final output.

The issue we have here is bucketing one related object's values, depending a separate value on a different related field.
The representation needs values from objects from different related fields, depending on what's what.
It's not particularly simple, but I can't imagine the use-case is too exotic either.

For our case, at the numbers we're looking at, doing the secondary mapping in Python is not a problem, but we initially looked at trying to do it in readers to push that work to the DB.

Given how far we've got, it's certainly possible to do much more complex things with readers, but it's a bit long-winded as we've discussed. Keeping the readers code simple looks significantly easier to deal medium term.

Nonetheless, I'd still like to keep experimenting here.

[j4mie]

Not really an answer as such, but here's a slightly higher-level way to achieve the thing you were trying to do:

spec = [
    ...
    {
        "owner_group": (
            pairs.discard_projector(
                specs.relationship(
                    "owner",
                    [{"group": ["name"]}],
                    to_attr="owner_for_owner_group",
                )
            ),
            projectors.producer_to_projector(
                "name", producers.attr("owner_for_owner_group.group.name")
            )
        )
    },
    ...
]

• We use a spec to express the relationship to avoid manually building the queryset function, including specifying a custom to_attr so we don't clash with the existing owner relationship in the spec.
• We discard the default projector that is returned from specs.relationship and substitute it with our own.
• We're exploiting the somewhat-undocumented fact that producers.attr behaves like operator.attrgetter in that it can follow dotted paths through related objects.

You could also just substitute the second part of the pair with lambda widget: {"name": widget.owner_for_owner_group.group.name} which might be clearer, but doesn't handle null relationships (whether or not this matters depends on your data model).

If we wanted to genericise this somehow and add it to the library, we'd have to consider:

• Composition: if you wanted to pull up (say) two fields from a related object to the root, these would both trigger a prefetch through the same relationship, which would cause an error. You'd have to either explicitly specify a different to_attr for both, or do something cleverer.
• Arbitrary complexity: you might want to make any sort of transformation to the projection, pulling up individual fields or subsets of objects, possibly aliasing fields, etc. Providing a half-baked solution here would probably not be a good idea.
• Not encouraging bad practice: we want to guide people into a pit of success. Enabling this sort of arbitrary transformation and reshaping of data might encourage bad patterns.
• Schema generation would break for these transformed fields and need to be overriden manually, which would be a pain if you started doing it a lot.

Still pondering...

[carltongibson]

Thanks @j4mie, that makes sense.

Let me play with this pattern. New toys! 🎉

I totally agree with all your points. (It makes no sense to make doing really silly things too easy!)

I, at least, am still discovering the lay of the land here, so who knows where we end up.

Thanks for your help! (Do follow up if another thought comes to you in the bath though!)