Adds scaffolding for user contributions (#350)

We want to enable users to contribute Hamilton dataflows. Data is the oil,
so code isn’t. Therefore we think we can help people get started with Hamilton
by enabling more contributions that enable people to get started quickly with
things off the shelf.

The design premise is to:
1. Have a separate package called `sf-hamilton-contrib` that houses these dataflows.
2. We then autogenerate docusaurus docs based on that module and expose it at hub.dagworks.io.
3. I have created github workflows/and adjusted existing ones to minimize CI usage.
4. This isn’t 100% complete, but I think it’s good enough to be merged into main.

Right now the main thing to note, is the pip install has to be updated with each new sf-hamilton-contrib release until it’s out of RC stage.

Otherwise we're working on a template to ensure things are standardized between dataflows. One major TODO is to figure out the unit testing story.


---- Squashed commits -- lots of them:
* Removes files from hub docs that aren't needed

To not confuse anyone this removes a few files that
get overwritten when docs are generated.

Adds to the changelog (+9 squashed commits)
Squashed commits:
[9b5262b] Changes prints to logging statements
[c6e338b] Fixes off by one
[29ef702] Changes commits to be served via static
[0aa1a2d] Fix no official case
[6c84b9b] Adds site map

And upgrades docusaurus.
[1d6bd3e] Adds google analytics to hub

So that we can understand usage.
[6b3a379] Adds telemetry

Fixes a few things too

This commit should be squashed at some point.
[b7e7f7e] Enables gh pages to be run off of main

This is required for things to only run if they are merged to main.
[74a8eee] Adds scaffolding for user contributions

We want to enable users to contribute Hamilton dataflows. Data is the oil,
so code isn’t. Therefore we think we can help people get started with Hamilton
by enabling more contributions that enable people to get started quickly with
things off the shelf.

The design premise is to:
1. Have a separate package called `sf-hamilton-contrib` that houses these dataflows.
2. We then autogenerate docusaurus docs based on that module and expose it at hub.dagworks.io.
3. I have created github workflows/and adjusted existing ones to minimize CI usage.
4. This isn’t 100% complete, but I think it’s good enough to be merged into main.

Cleaning up code a bit (+41 squashed commits)
Squashed commits:
[5aaf10b] changing base to see if it works with custom domain
[0ce0f8d] Fixes build
[a6e828e] Revert "Reverting sidebar"

This reverts commit d21065e.
[d21065e] Reverting sidebar
[2eff762] FIxing build
[9a5fbaf] Adds more to docs

Adds more structure.
[9867f5e] Adding more docs to contrib module
[fc35509] Adds templates for contrib

You can only have one default. So I decided to link from our default to
the specific one for now. Depending on volume we should switch things around.
[7dc35e2] Constrains build to contrib directory path
[2088037] wip
[a2715f1] Fixes so that pre-commit is run for everything
[58dd21a] Adds more to contrib/readme

should skip tests
[024364d] Fixes script to not error out on non-zero status
[e2c2e78] Adds start of readme

This should stop tests early too.
[2a32754] updates circleci to skip contrib changes
[7fbd056] Test didn't work github manages things
[c7ffdb9] Testing path
[5c780c7] Adds tagging
[0aeb353] reverting path back to what it was
[21f86b1] changes project root
[8ba794c] fixes workflow yaml
[8f2cf89] WIP adjust fetch depth
[953618a] Changes to include all commits
[cc56c44] Fix compile of docs path
[3e954fb] WIP fix python build
[edfe6f6] Adds compile step
[c5dd405] commit to test things
[b215379] Fixes path from moving package
[4826440] wip
[d415a37] WIP packages

Seems to work and gets added to the namespace appropriately without breaking
anything else.
[c1db95f] TODO: fix up package stuff.
[0ffc1b9] WIP docusaurus
[6d08fb7] wip
[1977375] WIP commit
[828060d] WIP fix github action flow
[7e5c3ab] WIP commit
[df5fd77] WIP
[4eb1333] wip
[721df5e] WIP
[9e918b3] wip
[9511774] WIP

Author: skrawcz

.circleci/config.yml

@@ -1,5 +1,26 @@
 version: 2.1
 jobs:
+  check_for_changes:
+    docker:
+      - image: circleci/python:3.10
+    steps:
+      - checkout
+      - run:
+          name: Check for changes in specific paths
+          command: |
+            set +e
+            git diff --name-only HEAD^ HEAD | grep '^.ci\|^.circleci\|^graph_adapter_tests\|^hamilton\|^plugin_tests\|^tests\|^requirements\|setup' > /dev/null
+            if [ $? -eq 0 ]; then
+              echo "Changes found in target paths."
+              echo 'true' > /tmp/changes_detected
+            else
+              echo "No changes found in target paths."
+              echo 'false' > /tmp/changes_detected
+            fi
+      - persist_to_workspace:
+          root: /tmp
+          paths:
+            - changes_detected
   test:
     parameters:
       python-version:
@@ -13,6 +34,15 @@ jobs:
       CI: true
     steps:
       - checkout
+      - attach_workspace:
+          at: /tmp
+      - run:
+          name: Check if changes were detected
+          command: |
+            if grep -q 'false' /tmp/changes_detected; then
+              echo "No changes detected, skipping job..."
+              circleci-agent step halt
+            fi
       - run:
           name: install dependencies
           command: .ci/setup.sh
@@ -22,19 +52,28 @@ jobs:
 workflows:
   unit-test-workflow:
     jobs:
+      - check_for_changes
       - test:
+          requires:
+            - check_for_changes
           name: build-py37
           python-version: '3.7'
           task: tests
       - test:
+          requires:
+            - check_for_changes
           name: build-py38
           python-version: '3.8'
           task: tests
       - test:
+          requires:
+            - check_for_changes
           name: build-py39
           python-version: '3.9'
           task: tests
       - test:
+          requires:
+            - check_for_changes
           name: build-py310
           python-version: '3.10'
           task: tests
@@ -47,22 +86,32 @@ workflows:
           python-version: '3.9'
           task: pre-commit
       - test:
+          requires:
+            - check_for_changes
           name: dask-py39
           python-version: '3.9'
           task: dask
       - test:
+          requires:
+            - check_for_changes
           name: ray-py39
           python-version: '3.9'
           task: ray
       - test:
+          requires:
+            - check_for_changes
           name: spark-py39
           python-version: '3.9'
           task: pyspark
       - test:
+          requires:
+            - check_for_changes
           name: spark-py310
           python-version: '3.10'
           task: pyspark
       - test:
+          requires:
+            - check_for_changes
           name: spark-py311
           python-version: '3.11'
           task: pyspark
@@ -71,30 +120,44 @@ workflows:
           python-version: '3.7'
           task: integrations
       - test:
+          requires:
+            - check_for_changes
           name: integrations-py38
           python-version: '3.8'
           task: integrations
       - test:
+          requires:
+            - check_for_changes
           name: integrations-py39
           python-version: '3.9'
           task: integrations
       - test:
+          requires:
+            - check_for_changes
           name: integrations-py310
           python-version: '3.10'
           task: integrations
       - test:
+          requires:
+            - check_for_changes
           name: integrations-py311
           python-version: '3.11'
           task: integrations
       - test:
+          requires:
+            - check_for_changes
           name: asyncio-py39
           python-version: '3.9'
           task: async
       - test:
+          requires:
+            - check_for_changes
           name: asyncio-py310
           python-version: '3.10'
           task: async
       - test:
+          requires:
+            - check_for_changes
           name: asyncio-py311
           python-version: '3.11'
           task: async

.github/HAMILTON_CONTRIB_PR_TEMPLATE.md

@@ -0,0 +1,32 @@
+[Summary of contribution]
+
+## For new dataflows:
+Do you have the following?
+- [ ] Added a directory mapping to my github user name in the contrib/hamilton/contrib/user directory.
+  - [ ] If my author names contains hyphens I have replaced them with underscores.
+  - [ ] If my author name starts with a number, I have prefixed it with an underscore.
+  - [ ] If your author name is a python reserved keyword. Reach out to the maintainers for help.
+  - [ ] Added an author.md file under my username directory and is filled out.
+  - [ ] Added an __init__.py file under my username directory.
+- [ ] Added a new folder for my dataflow under my username directory.
+  - [ ] Added a README.md file under my dataflow directory that follows the standard headings and is filled out.
+  - [ ] Added a __init__.py file under my dataflow directory that contains the Hamilton code.
+  - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Hamilton.
+  - [ ] Added tags.json under my dataflow directory to curate my dataflow.
+  - [ ] Added valid_configs.jsonl under my dataflow directory to specify the valid configurations.
+  - [ ] Added a dag.png that shows one possible configuration of my dataflow.
+
+## For existing dataflows -- what has changed?
+
+## How I tested this
+
+## Notes
+
+## Checklist
+
+- [ ] PR has an informative and human-readable title (this will be pulled into the release notes)
+- [ ] Changes are limited to a single goal (no scope creep)
+- [ ] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
+- [ ] Any _change_ in functionality is tested
+- [ ] New functions are documented (with a description, list of inputs, and expected output)
+- [ ] Dataflow documentation has been updated if adding/changing functionality.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,3 +1,9 @@
+Looking to submit a Hamilton Dataflow to the sf-hamilton-contrib module? If so go the the `Preview` tab and select the appropriate sub-template:
+* [sf-hamilton-contrib template](?expand=1&template=HAMILTON_CONTRIB_PR_TEMPLATE.md)
+
+Else remove this block.
+
+---
 [Short description explaining the high-level reason for the pull request]
 
 ## Changes

.github/workflows/docusaurus-gh-pages.yml

@@ -2,9 +2,11 @@
 name: Deploy Docusaurus with GitHub Pages dependencies preinstalled
 
 on:
-  # Runs on pushes targeting the default branch
+  # Runs on pushes targeting the default branch & contrib subdirectory
   push:
-    branches: ["user_contrib"]
+    branches: ["user_contrib", "main"]
+    paths:
+      - 'contrib/**'
 
   # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
@@ -30,16 +32,34 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v3
+        with:
+          fetch-depth: 1000
       # 👇 Build steps
+      - name: Set up Python 3.10
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      - name: Install dependencies
+        run: |
+          pip install -e .
+      - name: Compile code to create pages
+        working-directory: contrib/docs
+        run: python compile_docs.py
       - name: Set up Node.js
         uses: actions/setup-node@v3
         with:
-          path: contrib/docs
           node-version: 16.x
           cache: yarn
+          # The action defaults to search for the dependency file
+          # (package-lock.json or yarn.lock) in the repository root, and uses
+          # its hash as a part of the cache key.
+          # https://github.com/actions/setup-node#caching-packages-dependencies
+          cache-dependency-path: "./contrib/docs/package-lock.json"
       - name: Install dependencies
+        working-directory: contrib/docs
         run: yarn install --frozen-lockfile --non-interactive
       - name: Build
+        working-directory: contrib/docs
         run: yarn build
       # 👆 Build steps
       - name: Setup Pages

contrib/MANIFEST.in

@@ -0,0 +1,3 @@
+include *.md
+include *.txt
+include *.jsonl

contrib/README.md

@@ -0,0 +1,111 @@
+# Off-the-shelf Hamilton Dataflows
+
+Welcome!
+
+Here you'll find a package that curates a collection of Hamilton Dataflows that are
+ready to be used in your own projects. They are user-contributed and maintained, with
+the goal of making it easier for you to get started with Hamilton.
+
+We expect this collection to grow over time, so check back often! As dataflows become mature we
+will move them into the official sub-package of this respository and become maintained by the
+Hamilton team.
+
+## Usage
+There are two ways to get access to dataflows in this package. For either approach,
+the assumption is that you have the requisite python dependencies installed on your system.
+You'll get import errors if you don't. Don't know what you need, we have convenience functions to help!
+
+For more extensive documentation, please see [Hamilton User Contrib documentation]().
+
+### Static installation
+This approach relies on you installing the package on your system. This is the recommended path for
+production purposes as you can version-lock your dependencies.
+
+To install the package, run:
+
+```bash
+pip install sf-hamilton-contrib==0.0.1rc1
+```
+
+Once installed, you can import the dataflows as follows.
+
+Things you need to know:
+1. Whether it's a user or official dataflow. If user, what the name of the user is.
+2. The name of the dataflow.
+```python
+from hamilton import driver
+# from hamilton.contrib.official import NAME_OF_DATAFLOW
+from hamilton.contrib.user.NAME_OF_USER import NAME_OF_DATAFLOW
+
+dr = (
+    driver.Builder()
+    .with_config({})  # replace with configuration as appropriate
+    .with_modules(NAME_OF_DATAFLOW)
+    .build()
+)
+# execute the dataflow, specifying what you want back. Will return a dictionary.
+result = dr.execute(
+    [NAME_OF_DATAFLOW.FUNCTION_NAME, ...],  # this specifies what you want back
+    inputs={...}  # pass in inputs as appropriate
+)
+```
+
+### Dynamic installation
+Here we dynamically download the dataflow from the internet and execute it. This is useful for quickly
+iterating in a notebook and pulling in just the dataflow you need.
+
+```python
+from hamilton import dataflow, driver
+
+# downloads into ~/.hamilton/dataflows and loads the module -- WARNING: ensure you know what code you're importing!
+# NAME_OF_DATAFLOW = dataflow.import_module("NAME_OF_DATAFLOW") # if using official dataflow
+NAME_OF_DATAFLOW = dataflow.import_module("NAME_OF_DATAFLOW", "NAME_OF_USER")
+dr = (
+  driver.Builder()
+  .with_config({})  # replace with configuration as appropriate
+  .with_modules(NAME_OF_DATAFLOW)
+  .build()
+)
+# execute the dataflow, specifying what you want back. Will return a dictionary.
+result = dr.execute(
+  [NAME_OF_DATAFLOW.FUNCTION_NAME, ...],  # this specifies what you want back
+  inputs={...}  # pass in inputs as appropriate
+)
+```
+
+## How to contribute
+
+If you have a dataflow that you would like to share with the community, please submit a pull request
+to this repository. We will review your dataflow and if it meets our standards we will add it to the
+package. To submit a pull request please use [this link](TODO) as it'll take you to the specific PR template.
+
+### Dataflow standards
+We want to ensure that the dataflows in this package are of high quality and are easy to use. To that end,
+we have a set of standards that we expect all dataflows to meet. If you have any questions, please reach out.
+
+Standards:
+- The dataflow must be a valid Python module.
+- It must not do anything malicious.
+- It must be well documented.
+- It must work.
+- It must follow our standard structure as outlined below.
+
+
+### Checklist for new dataflows:
+Do you have the following?
+- [ ] Added a directory mapping to my github user name in the contrib/hamilton/contrib/user directory.
+  - [ ] If my author names contains hyphens I have replaced them with underscores.
+  - [ ] If my author name starts with a number, I have prefixed it with an underscore.
+  - [ ] If your author name is a python reserved keyword. Reach out to the maintainers for help.
+  - [ ] Added an author.md file under my username directory and is filled out.
+  - [ ] Added an __init__.py file under my username directory.
+- [ ] Added a new folder for my dataflow under my username directory.
+  - [ ] Added a README.md file under my dataflow directory that follows the standard headings and is filled out.
+  - [ ] Added a __init__.py file under my dataflow directory that contains the Hamilton code.
+  - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Hamilton.
+  - [ ] Added tags.json under my dataflow directory to curate my dataflow.
+  - [ ] Added valid_configs.jsonl under my dataflow directory to specify the valid configurations.
+  - [ ] Added a dag.png that shows one possible configuration of my dataflow.
+
+# Got questions?
+Join our [slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) community to chat/ask Qs/etc.

contrib/docs/.gitignore

@@ -0,0 +1,20 @@
+# Dependencies
+/node_modules
+
+# Production
+/build
+
+# Generated files
+.docusaurus
+.cache-loader
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*