Adapter Monorepo - RFC

[TillaTheHun0]

Currently our hyper adapters are stored in a single repo. With the move to deno in #230, we are considering the advantages of keeping all adapter code in a monorepo and whether or not we should split them out each into their own separate repo. As a proponent of the monorepo approach I hope to provide some reasons for keeping adapters in a single repo.

Firstly, a note on monorepos. JS monorepos, up until recently, has been synonymous with Node. Monorepos in Node typically necessitated more tooling due to challenges of managing dependencies via package.json and node_modules, hoisting from package managers like yarn and npm, dependencies between packages in the monorepo, and module resolution inherent with Node and node_modules. To be short, monorepos are hard in Node, because Node dependency management makes it hard.

With Deno, all of these issues are moot, because dependency management is inherently different from Node. Because any file can be imported from any relative path or URL, is pulled at runtime, and with no node_modules shimming and magic ✨ , Deno dependency management works no differently in a monorepo. In fact, Deno monorepo is being used to great effect by the Deno core team and the std lib. Due to Deno's dependency system, Deno projects tend to be flatter than node projects which I believe compliments monorepo and results in great DX.

With that out of the way, here are some monorepo benefits for maintainability and DX

I propose a structure along the lines of:

adapter-couchdb/
adapter-fs/
adapter-memory/
...
Makefile
lint-staged.config.js

Tooling

Scripts that enforce things like commit linting, code linting and formatting, test coverage, can all be written once, stored beside the code, instead of duplicated across repos.

One repo to clone

Maintainer do not have to clone many repos and reference across repos. Contributing to hyper is simply cloning a single repo. Part of adapter development will undoubtedly be using other adapters for reference. This becomes trivial with monorepo.

One repo to commit

adapters that change together, something as trivial as code styling changes, or even something like an api change, can be done atomically, with a single commit to one repo.

One repo to release

Deno releases works off of git tags and hooks. By ensuring our commit messages are linted, we can use tools to automate semver bumping and tagging commits, per changed adapter, and changelog generation. Tools like lerna can be pulled in via skypack to make releasing multiple adapters a simple Github workflow. Even without lerna, this can be built and maintained alongside the adapter code in the monorepo.

[TillaTheHun0]

I believe regardless of monorepo, we may need to use tools from node ecosystem like husky and lint-staged and commitlint, at least until the deno ecosystem matures more.

[twilson63]

• husky just generates git hooks right? Why don't we just create git hooks with a shell script?
• what does lint-stagged do?
• what does commitlint do?

[TillaTheHun0]

Indeed, husky just creates git hooks, with a little extra niceties in terms of the api. We could replicate it.

lint-staged is used to specify quality checking rules on git staged files. So doing things like running tests, linting, formatting, etc.

commitlint enforces the commit message style. This is crucial is want things like automatic changelog generation, and automatic semver bumps and tagging.

[twilson63]

Pros to keeping adapters separate

Adapters should have separate versions driven by git-tags not the same version as the core

By separating adapters into individual repos, they can have their own semantic version and generate clear guidance to any user of the adapter by the version number major.minor.patch - this version number is specific to the adapter and informs the user of the hyper adapter what their version state is to the version state of the current adapter. hyper has no control of services it wraps adapters around and as they change, there may be multiple production versions that may need to be supported. It would be more important to have clarity when adapter-redis changes from version 1.x to version 2.x that 1.x supports redis 6 while 2.x supports redis 7 - This way if the user of hyper is using redis 6 they know they should be using v 1.x of hyper-adapter-redis and if they are using redis 7 then they should be using hyper-adapter-redis v2.x

Small and less confusing code base will encourage many developers to contribute

I think contributors have less anxiety contributing to smaller focused code bases than large confusing code bases, I think many contributors will be part of a service eco-system and will want to build an adapter for hyper to connect with their service versus a hyper contributor that wants wrap a specific service. So targeting contributors with an adapter template and maintaining adapters abstracted from core makes more sense.

They simply don't need to understand all the moving parts of hyper to contribute, they just need to understand the mechanics of the adapter and the service it is wrapping.

Separation of concerns

Adapter modifications rarely cause changes to other parts of hyper and changes to other parts of hyper rarely cause changes to adapters

Since working with hyper for the last 4 months, many change requests revolve around specific adapter implementations and very rarely around modifications core, app or port. Having adapters in separate focused repo's will make the process to create frictionless lifecycle process around adapters will encourage focus on the quality and reliability of the adapter. The current monorepo complexity has removed that focus on quality of the adapter.

As an adapter contributor, I want to focus on the adapter and the service of that adapter with the ability to spin up a service instance locally and run tests against that service instance as I development to make sure the adapter is properly handling the requests and to isolate issues between the adapter layer and the service layer.

I am more inclined to work and focus on the adapter in an isolated repository that has a separate lifecycle of deployment and release management than that of hyper core.

Continuous Delivery

Release management should be independent

hyper is not a runtime like deno or node, hyper adapter releases should be published when they are ready not batched up into release candidates with multiple changes. hyper adapters should release independently of other hyper adapters, core should release independently of adapters and core should always be backwards compatible with any hyper adapter so that concurrent releases can occur without the dependency of the other. This is a loosely coupled architecture. Deno and Deno's std library are not loosely coupled they are tightly coupled which means the changes in the deno runtime effects the std library and vice versa so it makes sense to the deno team to manage these in a single repo, but an adapter would be more like a deno third party module, for example opine this module is not part of standard because it is opinionated and focuses on a specific use case.

Adapters in hyper are opinionated and focus on specific service, where hyper core does not care about any service, it just cares about the ports or interfaces.

Visibility

Clear Visibility for adapter users and contributors

Adapters as separate repositories can be much easier to find and connect the dots from searching in a standard library, navigating to the github repo. It is clear to the potential contributor or user of the adapter how to access the source code and it is not buried in a complex repository. Adapter users will be more likely to report bugs as issues on the actual adapter providing better commit history and clear communication to the user when a problem with their use of the adapter is resolved. I think the mono repo could create a lot of noise for users trying to find out if their issue is addressed or not, also will create a lot of work on the core maintainers to parse and batch releases, which goes against continuous delivery practices.

Consistency

Embedded adapters and third party adapters will create a lack of consistency that will confuse adapter users

I think consistency is very important and it could be confusing for users of hyper to see adapters in separate repos and adapters embedded within hyper core. Should we aim to be consistent and either expect contributors to build adapters in core which is a clear source of confusion. or provide a clean guide and template repo for contributors to build adapters in separate repos? That is the real question in my mind, it seems from our recent interview to have an isolated template repo will encourage more contribution than the fear of breaking the mono repo or the uncerntainy around the release of the adapter because of batched releases.

Summary

To me it comes down to two choices, to we focus streamlining the process for adapter contribution and adapter usage so it is frictionless for our users and contributors. Or we choose to make the maintenance of adapters frictionless for the hyper core team. As a goal to keep core team lean, I am concerned this will lead to more core ownership of adapters than intended, because it draws a line between owner and guest. Even if the contributor of the adapter was not part of the core team, I think they will feel like their contribution is a handoff instead of a collaboration. I would recommend we decide leveraging inclusive guidance and encouraging contributors to take full ownership of their contribution instead of a hand off that becomes the responsibility of the core team to manage and support. If we only plan to have 10 adapters this may not be a problem, but I think we could have 100's of adapters and more than 10 ports for hyper. I prefer the core team focus on ports, core, interface, and let the adapters be managed by contributors whose knowledge is focused around the adapter service, not the inner workings of hyper core.

[twilson63]

One note of history, since the pouchdb plugin system went to mono repo, the contribution levels have decreased and the releases and moved to 18 month cycle. Now that may be because of maturity, or lack of adoption, but since the consolidation the number of contributions have dropped.

Do we know of any eco system where the switch to mono repos has increased contributor engagement? and not increased core team's responsibility? It would be good to reference such an eco system.

Do not include deno or node they are different use cases and I don't think provide much comparison value to this discussion.

[twilson63]

Final Decision

All in all, I default the final decison to @TillaTheHun0 and @tripott you guys should decide, you have heard my arguments, I feel the app, core and ports should be under one repo and think ports should just be a part of core, and maybe app part of core too with GQL and REST combined. And adapters should be external and community encouraged. I also think it will be easier to contract out the creation of adapters and even the support of adapters in separate repositories than in the core hyper repo. I feel the lack of contribution has not only been lack of documentation, but the fear of breaking changes, no one wants to be the guy who broke the build and the more complex the core repo becomes the more real this becomes an issue for new contributors.

Maybe you guys can huddle the next two days and make a decision, I will support it either way, just know that this is a decision that we will have to look back in 2 years and know whether we made the right call or not. So think in terms of having 100s of adapters and a dozen or so ports from sms, image recon, sentiment analysis, blockchain, etc. Look at open source communities that support independent plugin contributions like rollup, babel, webpack and eco systems that have mono repo plugin contributions like pouchdb, ?

mono repos are cool, but they are in eco-systems that pre-built tooling goes against the grain. Google, Facebook etc have 1000s of engineers and have the internal expertise to leverage mono-repos effectively, and do not use mono-repos for their open source pojects, mainly they have mono-repos (to my knowledge) as internal projects, it is a way of protecting IP and enabling cross team contribution and communication. I don't think we have the same issues of a Google or Facebook. Do we have any successful evidence of three person teams building open-source leveraging the pros of a mono-repo?

Please give all the above consideration, try to not only see the present, but the future, and our desire to stay lean and empower the community to grow the project, instead of having a core team of 20 developers managing hundreds of adapters. Don't get me wrong it will be great to see 100s of adapters, but managed by the open source community not the hyper core team. And I think we can find a way to compensate owners of adapters by the community in a marketplace model to fund the support of the adapters which is not free. And my intentions are not to make it free, as much as to separate adapter knowledge from hyper core knowledge.

Ok, end rant, you guys make the call, and I will support your decision.

[TillaTheHun0]

Thanks Tom for raising those concerns. Though I feel some of those concerns are not what i'm advocating for, when I advocate for monorepo.

Adapters should have separate versions driven by git-tags not the same version as the core

I absolutely agree. Is your understanding that monorepo doesn't allow for this? As I've said before, version is a property of code, not a property of source control. There are many ecosystems where packages are semver bumped and released separately from one another. Monorepo goes a step further and allows for those independent bumps and releases to be driven from a single atomic commit. Consider:

commit-1234abcd contains changes to package [email protected] and package [email protected]
ci bumps A to 1.4.0 and git tags commit-1234abcd with tag "[email protected]"
ci bumps B to 2.2.0 and git tags commit-1234abcd with tag "[email protected]"
ci releases new version of A
ci releases B new version of B

I am not advocating for monolithic release cycle. In fact, a monolith is what i've seen come from having multiple git repos. When changes need to be made across multiple repos and released in tandem, you've no choice but to coordinate when things get bumped and released. This is difficult to do across repos with governance; welcome to a monolith. Monorepo allows for doing this independent release + atomic commit, seamlessly.

An example of this is Apollo's monorepo for apollo-server. Packages are stored together and can be changed together, but are versioned and released independently from one another.

Small and less confusing code base will encourage many developers to contribute

I can see this. Though I think this can be solved with tooling and documentation. Also there will undoubtedly be separate repos outside hyper org for community driven adapters that the core team does not maintain. The monorepo would only contain what the core team is obligated to maintain. Having a single repo makes it much easier for core team to monitor and service OSS contributions rather than having dozens of repos to keep track of issues and PRs.

An example of this is Rollup plugins. All official Rollup plugins are kept in a single repo. Unofficial, community driven plugins, are kept in their own separate repo

Separation of concerns
I am more inclined to work and focus on the adapter in an isolated repository that has a separate lifecycle of deployment and release management than that of hyper core.

Nothing is deployed or released together unless we want it to be. Monorepo is just storing code together. Though I can see a case for core and ports and apps in a repo, and then all adapters together in a separate repo.

Continuous Delivery

I feel i've already addressed this. Storing code together does not mean releasing it as a monolithic version or artifact. Multiple artifacts can be generated from a single repo. See my example above.

Visibility

i'm not convinced that monorepo makes things less visible. If anything, I think it makes things more visible. I can see all issues and PRs and close and link them in a single place, from a single commit history instead of jumping back and forth between repos.

Consistency

it sounds like the concern here is storing adapters with core, even if they are versioned separately in the same repo. I am fine with breaking out adapters together into a hyper/adapters repo, similar to rollup. A that point though, I think monorepo is more conducive for consistency than separate repos, because tooling can be shared and conventions can be enforced in a single place. Separate repos diverge over time, and the only way i've seen them prevented from diverging is through governance which just slows down decisions instead of addressing the core issue.

---

If you're asking for examples of monorepos for projects with large open source contributions, particularly ones with any sort of plugin/adapter feature, I can name a few:

Rollup plugins
Webpack plugins
Babel Plugins
Apollo Server
Vercel
Next plugins

Each of these have plugin architectures where core team helps maintain official plugins and then separate repos are spun up and maintained by the community for community driven plugins.

Conclusion

I think monorepo for adapters is the way to go, but also think the discussion might be moot because of the way deno land releases entire repos based solely off of git tags.

I don't like that approach. It places too much importance on the source control mechanism for releasing code. I should be able to move code between repos, or even choose different source control and not lose my version. Nest.land doesn't do this. It allows for specifying which files to publish and keeps version stored with the code.

Perhaps Deno.land does this because of a reason i'm not understanding. Maybe I have something akin to "node blinders" 😆 .

For now, I think we can split things into separate repos, but centralize tooling. Then we can just use Deno's url importing, in each adapter repo, to share the tooling. Let's see how it goes.

[tripott]

I think adapters should be external to promote open source contribution.

[TillaTheHun0]

We will split each adapter into a separate repo in the hyper org.