Refactor of Environment

[tomjn]

I propose the following:

• erase the current provisioners
• provision a single install with a single plugin folder
• import a pre-made database
• use the git repos instead of svn, composer were possible

.org is a multi-network multisite, it would be easier if we just leaned into that. This eliminates a lot of the complexity, as well as a Windows compatibility issue ( symlinks require admin privileges ), and brings it a little closer to production

Since this repo was created, VVV itself has moved forwards, we now have a folders: key, so in theory you could reproduce a single .org site using just config.yml and the standard site if you knew which repos go where

[iandunn]

Do you mean a single install/plugins/network folder for wordpress.org, and then other installs for wordcamp.org, jobs.wordpress.net, etc; or do you mean a single multi-network for all the sites, even the ones that aren't part of the production wordpress.org multi-network?

[tomjn]

I had no idea that those were completely separate installations, do you have information on how production is structured and which sites are on which installs?

[iandunn]

The current provisioners are setup pretty close to production.

• wordpress.org physical servers
  • wordpress.org multi-network multi-site install
    • network 1: wordpress.org/, wordpress.org/news, wordpress.org/plugins, login.wordpress.org, wordpress.org/five-for-the-future, etc
    • network 2: make.wordpress.org/, make.wordpress.org/core, etc
    • network 3: learn.wordpress.org/
    • network 4: unused, old experiment migrating wordcamp.org to this codebase
    • network 5: developer.wordpress.org/
    • network 6: global.wordpress.org/, es.wordpress.org, etc
    • network 7: translators.wordpress.org
    • network 8: livestream.wordpress.org
  • api.wordpress.org - not in wordpress, mostly raw php files. uses the same db cluster, though, and sometimes loads WP to process requests, because they're on the same physical filesystem
  • buddypress multisite - buddypress.org/, bbpress.org, profiles.wordpress.org/, etc
• wordcamp.org physical server - multisite installation
• wordpress.net physical server - separate/isolated single-site installations
  • jobs.wordpress.net
  • wp15.wordpress.net
  • etc
• wordpress.com servers - wordpress.tv, blog.wordpress.tv, status.wordpress.org, etc

There's lots more sites and some other complexity -- we've been around for 20 years after all -- but that's the gist of it.

It doesn't seem like it'd be simple to combine all those codebases into a single multi-network install, and still stay close enough to production to be a reliable match. For example, wordpress.org and wordcamp.org both have sunrise.php files. There are probably some places on w.org where network and site IDs are hardcoded, etc.

I could be wrong, though, and we could make some intentional breaks from production in order to make contributing easier. Committers always have their w.org sandboxes to use as staging environments before deploying.

[tomjn]

Then how about this:

1. we separate out each environment into its own repo, much how wordcamp.org is now in its own repository
2. the repo includes an SQL file for initial test data
3. we then let people choose their own local environment and make no assumptions about it locally

This way it's super easy to setup with MAMP or another native setup. We can use WP Env via wp-env start and a .wp-env.json file, or just clone it into an existing site via one of the many dev options.

This way everybody gets to use what they want to use, they don't have to provision all the things, most if not all the maintenance scripts can be eliminated and offloaded, etc

[dd32]

Then how about this:

That's actually pretty close to what is happening already, just not really documented/communicated as such. WordCamp.org has it's own docker setup, learn.wordpress.org has it's own wp-env environment, the Pattern Directory has it's own wp-env, some work-in-progress of migrating the Theme Directory over to wp-env, and I know a few others are in the works as contributions happen. If there's anything you'd like to contribute specifically @tomjn I'd probably just work on a self-contained wp-env setup for it (if it doesn't yet exist), noting that wp-env has some significant shortcomings in more complex setups (that's a tradeoff for simplicity in anyone being able to use it)

The biggest issue is the interconnected-ness of WordPress.org, and reliance upon other systems, Getting all the sites working "like production" is much harder than one would anticipate at times. Getting just the theme running without the rest of the functionality is easier though.

• Plugin Directory expects access to multiple SVNs, Support Forums, Cavalcade, a few other internal-only projects, and a GlotPress instance for full functionality.
• Theme Directory needs the support forums, and a SVN/Trac instance for most features and to top it off, it's UI is tied to api.wordpress.org (currently)

Having everything in a mono-repo also makes it harder, as using git means pulling the entirety of the meta repo and only using two folders out of a million (which is why Learn and WordCamp.org are easier being self-contained).
Migrating everything to individual Git Repo's and then using submodules will somewhat help there, but it's a looong road to do that with a lot of hurdles that would need to be jumped over before that could happen.

[tomjn]

A QQ, most of those have a .wp-env subfolder with an mu-plugins PHP file that gets mapped into place. Why not just place the file in the right location in the repository like the other mu-plugins files already in the repos? What problem is it avoiding 🤔 The rest of it appears on the surface to be portable enough for use outside of WP Env

[dd32]

most of those have a .wp-env subfolder with an mu-plugins PHP file that gets mapped into place. Why not just place the file in the right location in the repository like the other mu-plugins files already in the repos?

The main thing it's solving is that the mu-plugin in the local env repo is something that shouldn't be used in production. Ideally those mu-plugins wouldn't be required at all. It's usually stubbing other production plugins or something else that's not open-source
At least in the case of the theme directory one, I'm aiming to remove the mu-plugins, but it got things working faster to just include it and then slowly chip away at the main plugin/themes to remove the dependancy and clean things up properly.

[tomjn]

Currently the sites can be turned on or off via custom options in config.yml, I think it's worth looking into adding new options that clone the other repos and do the additional step to run the scripts already present so that work on one directly benefits the other.

This way existing users can change an option in their config.yml, and their existing sites remain. We deprecate those sites and update the instructions. They can turn on the new WordCamp site and reprovision and it will clone it and it'll be just like running WP Env ( perhaps it will run WP Env ). Thoughts?