Proposal: Use DB migrations for keeping the database up to date.

[StevenDufresne]

Goals

• Make it easier to track database changes
• Make it easier to make database updates (PRs)

Summary

As opposed to using a database snapshot, write migrations to create new tables and insert test environment data.

Pros

• Don't have to deal with data scrubbing
• Traceable DB timeline
• Easier to add PR's to fill gaps in data
  • smaller, easy to read diffs

Cons

• From a data standpoint, local environment will continually diverge from production.
  • Specifically when it comes to posts, media, etc ...
• Third Party Plugin configurations?
• Could potential overwrite custom configurations
• A few more steps
  • We may export some data from production, copy paste out some of the inserts into another sql file

Option 1 (My Preference)

Plugin Driven

We can write a plugin to manage the database update. A plugin would be useful because it won't be coupled to provisioning which we have identified we want to revisit. This will also keep the codebase in a structure & language the community understands... and can help contribute.

The plugin should:

• Fetch the migrations from a single source of truth (code repo -> Github/svn)
• Loop through and run the migrations using wpdb

Potential Folder Structure for Repo

/migrations
└── wordpress.org
  └── 2020-02-15.sql
  └── 2020-03-19.sql
  └── 2020-04-30.sql
└── wordcamp.org

We could also move the migrations near each code base as well by having them in the same wp-content folder as the site.

Option 2

Update VVV to run migrations

Modify the current .sh file to run the migrations as a new step in the process.

How can we get started?

Regardless of the approach, I think we could get started by pulling out the sql insert statements from the current db snapshot into a sql file which we would consider the first migration.

[iandunn]

From a data standpoint, local environment will continually diverge from production.

I'm not too worried about that; I think a dev should commit a migration for anything they want to keep.

The big downside IMO is the extra work it involves, but I don't see a better way. Maybe we could build a tool that would automate part of the work to create a migration, but that'd be a future iteration.

[iandunn]

I love the idea, but I think we may need to solve some other problems first.

I think the Meta Environment was a good step forward from where we were when it was created, but it hasn't really been successful. I think the main reason for that is that the majority of Meta committers (myself included) don't use it as their daily environment, so they don't keep it in sync with production, fix problems with it, etc. It's "only" a tool to help non-committers, which makes it extra work on top of our already packed schedules.

I think we need to consider switching to a model where committers and non-committers use the same environment. The could mean pivoting the Meta Environment, but could also mean deprecating it in favor of something new.

It'd probably be good to decide that before we invest a lot of time/energy in something like this.

[StevenDufresne]

I think the main reason for that is that the majority of Meta committers (myself included) don't use it as their daily environment

Great point!

If you find some time, do you mind listing some of the short comings here? Specifically, why don't you use the meta environment?

[iandunn]

There's probably a few more that aren't jumping out at me, but:

• Vagrant is slow
• Only one HTTP server can use port 80, so working on non-WME sites requires shutting down Vagrant, then booting up whatever container the other site uses. That's a slow and annoying process. i could work around that by using alternate ports and leaving all the containers running, but that feels awkward, and would probably also have performance issues.
• The ME doesn't have closed-source code, so it's impossible to work on many things. The % of closed code is much smaller now than it was a few years ago, but it's still an issue. The WordCamp Docker environment gets around that by having an extra step that committers can do to add the closed-source code. We could probably add something like that here too, if we wanted to commit to this.

For the past few years I've just been using a local MEMP setup, where I can have everything running at once, without any performance issues.

[tomjn]

Vagrant is slow

The meta environment provisioner is unnecessarily complex, which is why it's slow. Otherwise performance is rather good, and on Windows it's identical to Docker when using Hyper-V ( since Docker uses a Hyper-V VM running Alpine Linux as a container host ).

I would also note that WP Env and Docker require Windows 10 Pro/Enterprise. Windows 10 Home users would be excluded from contributing if WP Env was adapted.

Likewise, the WordCamp docker image is very tightly coupled in the way it's built, making the setup scripts non-portable. Using them in VVV/WSL/other containers/natively is a non-starter

Only one HTTP server can use port 80, so working on non-WME sites requires shutting down Vagrant, then booting up whatever container the other site uses. That's a slow and annoying process. i could work around that by using alternate ports and leaving all the containers running, but that feels awkward, and would probably also have performance issues.

Note that you can vagrant suspend and vagrant resume. I wish you had mentioned this to the VVV team, there are solutions and avenues of exploration.

The ME doesn't have closed-source code, so it's impossible to work on many things. The % of closed code is much smaller now than it was a few years ago, but it's still an issue. The WordCamp Docker environment gets around that by having an extra step that committers can do to add the closed-source code. We could probably add something like that here too, if we wanted to commit to this.

There are plugins in the meta environment that aren't on production and vice versa. Closed source isn't so much a concern as just making sure the plugins and versions are accurate. I don't see why automation couldn't solve this

[iandunn]

The meta environment provisioner is unnecessarily complex, which is why it's slow.

I was referring to things like TTFB, but that may have improved in the past few years. It'd be great to speed up the provisioner as well, though.

Windows 10 Home users would be excluded from contributing if WP Env was adapted.

That's a great point.

you can vagrant suspend and vagrant resume

That wouldn't solve the UX issue for me, but you mentioned Traefik elsewhere, which might. Although then I'd still have to have everything running at the same time. Maybe I'd just end up using the provision scripts and database, and running my own MEMP for the rest.

That's just me, though, and most folks would probably want a container, including other committers. As long as most committers were using the container, and almost all committers were using the provisioning & db scripts, then that could work.

If you think it's possible to solve the above problems w/ a refactor, then a new issue to discuss the details would be great!

If that's solved, we could circle back here to discuss better ways to keep the sample db up to date.

[tomjn]

I'd also note that Vagrant VMs don't use 127.0.0.1 and have their own IP, so port 80 shouldn't be a concern. I'm writing up an issue for refactoring the script

[pattonwebz]

I would ideally prefer a solution that is environment independent so anyone could choose to use VVV, MEMP, XAMPP or whatever they like. I personally prefer a docker env and do not currently have Vagrant installed on my system. I understand that moving away from VVV for this might be vastly out of scope though so having to install VVV to get it working would be acceptable for me.

I am afraid I do not have enough insight into how the current setup works or why it would never work for me. Primarily I was lacking essential data and I have a feeling that some of the systems likely relies on things in the closed source part of the .org site which I do not have access to.

Happy to be a tester for any changes proposed that could ease the path to getting this running for others :)

[iandunn]

@pattonwebz , I think @dd32 has tested out a Docker env for the theme directory specifically. That might help in the meantime, and could provide provision scripts, etc that WME could adapt in the future, if we decide to keep it going.

[SergeyBiryukov]

I would also note that WP Env and Docker require Windows 10 Pro/Enterprise. Windows 10 Home users would be excluded from contributing if WP Env was adapted.

Just noting that it is now possible to install Docker on Windows Home.

[SergeyBiryukov]

I would ideally prefer a solution that is environment independent so anyone could choose to use VVV, MEMP, XAMPP or whatever they like.

Same here, I use a custom WAMP stack and would ideally like to use it with Meta Environment too, if possible :) Though I guess a Docker environment might also work for me.