Explore a WP entity export iterator

[brandonpayton]

Motivation for the change, related issues

For data liberation, we want an API for streaming WP entities from a site. Then we can export WP entities to multiple targets without having to solve the same entity traversal problems in each exporter.

Related to #2106

Implementation details

TBD

Testing Instructions (or ideally a Blueprint)

TBD

[brandonpayton]

I've been reviewing WP tables and data structures to see what seems to make sense. This is a very rough outline that I plan to fill in and explore more tomorrow.

So far, the rough approach is to iterate over various entity iterators, most of which will be iterating over table rows with a condition like WHERE ID > previous_entity_ID ORDER BY ID. For the sake of performance we may select in chunks but still relay one entity at a time through the interface.

One possibly controversial direction so far is that I am thinking it may make sense to directly convey terms, taxonomy, and term relationships separately rather than modeling categories and tags as first class entities. Intuitively, it less fraught to just convey what is there and leave meaning-making to API consumers. Do you have any thoughts on this, @adamziel and @zaerl?

[brandonpayton]

Note: Some of the inline TODOs share some of my thinking.

[zaerl]

One possibly controversial direction so far is that I am thinking it may make sense to directly convey terms, taxonomy, and term relationships separately rather than modeling categories and tags as first-class entities

This is okay, and it's not controversial at all. But remember that the tables may have some artifacts and some special rules (example: a post is_sticky if is_sticky( $post->ID ) is true). The standard WXR exporter export_wp uses get_{categories|etc} functions that clean away data using the *_exists family of functions. The wp_term_relationships table exists only because we have many-to-many relationships with posts, and it can have "relationships" that do not exist anymore or refer to entities that do not exist anymore.

We should have two base XML exporters here and start from the simpler ones.

1. Having an exporter that copies the database by looping the rows, creating the XML one after another. Fast and not memory-hungry for obvious reasons.
2. Another should export with the 1:1 results of what core does. A WXR created by the core exporter guarantees you to have a specific structure: site options, terms ordered by hierarchies, and all the items after.

For the sake of performance we may select in chunks but still relay one entity at a time through the interface.

Reading the steps in twenty posts is okay. For every post, you should read the following:

1. The post meta(s)
2. The comments
   • The comment meta(s)

[brandonpayton]

Thank you for your feedback on this, @zaerl! It's helpful.

Having an exporter that copies the database by looping the rows, creating the XML one after another. Fast and not memory-hungry for obvious reasons.

I am working on this first. Currently, there is just a dumb iteration over database rows starting with terms tables, but it's going to have to be a bit smarter than that (I think). And the above provides a good context / test case to see whether this is headed in a reasonable direction.

[zaerl]

And the above provides a good context / test case to see whether this is headed in a reasonable direction.

Having both cases can be a good thing for us. The core export_wp function does, for example, use the can_export arg to check post types. The function core code is good! It does a lot of low-level queries, and the terms are the only ones that use the get_* style. To put categories in order with no child going before its parent. But this is no longer a problem with the importer.

Starting with the DB SQL queries is perfectly fine, and you made the right choice. 👍