Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector publishing automation #197

Merged
merged 121 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from 109 commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
402623f
Adding registry automation
sordina Apr 15, 2024
4c6abfa
check for first commit
sordina Apr 15, 2024
fd4d8d3
different listing logic
sordina Apr 15, 2024
57c84db
testing logic for changed files
sordina Apr 15, 2024
eb2a54e
depth
sordina Apr 16, 2024
c10217f
pseudocode
sordina Apr 16, 2024
63be059
quotes
sordina Apr 16, 2024
929ca8b
paths
sordina Apr 16, 2024
5c9d22b
readme update for testing purposes
sordina Apr 16, 2024
5d26c80
readme update for testing purposes
sordina Apr 16, 2024
a8341b4
comments
sordina Apr 16, 2024
b4cba7d
testing registry automation go
sordina Apr 29, 2024
cdb5c54
Merge branch 'main' into lyndon/registry-automation
sordina Apr 29, 2024
7f4c2a6
fixing type
sordina Apr 29, 2024
9cb0a80
readme
sordina Apr 29, 2024
8b1fade
sg test change
sordina Apr 29, 2024
589ea1e
WIP
sordina May 1, 2024
7f08397
Merge branch 'main' into lyndon/registry-automation
codingkarthik Jul 10, 2024
b737f20
use tj-actions/changed-files to get the list of changed files
codingkarthik Jul 10, 2024
9474d89
add a mock test package for azure cosmos
codingkarthik Jul 11, 2024
82ad61a
add a sample test registry automation
codingkarthik Jul 11, 2024
a14b419
print changed files using env vars
codingkarthik Jul 11, 2024
b2cc717
add fetch depth 1
codingkarthik Jul 11, 2024
89ba355
cat the added files
codingkarthik Jul 11, 2024
55ae384
Just get registry files
codingkarthik Jul 11, 2024
71535a1
registry /*
codingkarthik Jul 11, 2024
82efb8c
print all changed files
codingkarthik Jul 11, 2024
15933ca
remove toJson
codingkarthik Jul 11, 2024
d92c5a7
remove toJson
codingkarthik Jul 11, 2024
55ae345
print out added and modified files separately
codingkarthik Jul 11, 2024
1c370ab
remove `json: true`
codingkarthik Jul 11, 2024
fbd3c1b
add \n separator
codingkarthik Jul 11, 2024
cf5161c
print out DELETED files
codingkarthik Jul 11, 2024
ed20882
use deleted files
codingkarthik Jul 11, 2024
8f98cd2
add azure-cosmos logo back
codingkarthik Jul 11, 2024
77d51d1
delete azure cosmos logo
codingkarthik Jul 11, 2024
6666be3
delete registry-automation-test logo.png
codingkarthik Jul 11, 2024
575676d
add back the deleted logo
codingkarthik Jul 11, 2024
11333b4
construct JSON object
codingkarthik Jul 11, 2024
5162914
modify the jq command
codingkarthik Jul 11, 2024
c409d7d
another way of doing jq things
codingkarthik Jul 11, 2024
0c0fa05
try printing the output
codingkarthik Jul 11, 2024
8ea30f4
unstringify json
codingkarthik Jul 11, 2024
85b9a0d
vanilla jq
codingkarthik Jul 11, 2024
8766377
recursively unstrigify object and array JSON
codingkarthik Jul 11, 2024
87a7267
echo all the added, modified and deleted files
codingkarthik Jul 16, 2024
51c35b4
remove redundant double quotes
codingkarthik Jul 16, 2024
059c766
write changed files output to a file
codingkarthik Jul 16, 2024
92942bf
try removing toJson
codingkarthik Jul 16, 2024
67202ff
try fromJSON
codingkarthik Jul 16, 2024
b3521ad
use string instead of toJson
codingkarthik Jul 16, 2024
435c203
Merge branch 'main' into kc/test-registry-automation
codingkarthik Jul 16, 2024
b069959
use toJson only
codingkarthik Jul 16, 2024
75a6239
create a JSON object
codingkarthik Jul 16, 2024
da932a7
print the content of the changed files
codingkarthik Jul 16, 2024
bb0a939
remove double quotes from the array values
codingkarthik Jul 16, 2024
9fd2eb0
remove jq
codingkarthik Jul 16, 2024
42d9c9c
Parse the changed files JSON file
codingkarthik Jul 16, 2024
332366f
remove the CONNECTOR_HUB_DIRECTORY flag
codingkarthik Jul 16, 2024
92619ff
move the file to registry-automation first
codingkarthik Jul 16, 2024
dffcf74
use io.ReadAll
codingkarthik Jul 16, 2024
2fbb3ba
print out the byte value
codingkarthik Jul 16, 2024
4a3eb75
convert into string first
codingkarthik Jul 16, 2024
47d937a
don't escape json
codingkarthik Jul 16, 2024
0ed25e5
delete azure cosmos logo
codingkarthik Jul 16, 2024
968ce1a
remove logo from registry-automation-test
codingkarthik Jul 16, 2024
76370c3
modify the registry-updates workflow
codingkarthik Jul 17, 2024
35866dd
use jq to print out stuff
codingkarthik Jul 17, 2024
4482ed3
separate out connector version changes from metadata changes
codingkarthik Jul 17, 2024
baf9bd8
pattern match on the registry metadata files
codingkarthik Jul 17, 2024
4116d6e
use files_yaml instead of files
codingkarthik Jul 17, 2024
cfd3043
use cat instead of jq
codingkarthik Jul 17, 2024
fcb1ca8
rename files according to new format
codingkarthik Jul 18, 2024
6d709e1
modify the regex
codingkarthik Jul 18, 2024
b2461ce
fix the regex
codingkarthik Jul 18, 2024
6cc7dd1
WIP commit to build the registry payload
codingkarthik Jul 25, 2024
71717ab
Build out the registry payload
codingkarthik Jul 25, 2024
d795855
collect all the processed connectors
codingkarthik Jul 29, 2024
8882013
no-op refactors
codingkarthik Jul 30, 2024
fd32b0c
remove redundant fields
codingkarthik Jul 30, 2024
1ce8713
minor no-op
codingkarthik Jul 31, 2024
f2928f6
WIP: gather CLI args in a struct
codingkarthik Jul 31, 2024
15537e9
include connector's namespace in the registry path
codingkarthik Jul 31, 2024
e368a05
add a debug log in cleaning uploaded connector versions
codingkarthik Jul 31, 2024
bf22df2
remove stray comment
codingkarthik Jul 31, 2024
359f473
use the GCP_SERVICE_ACCOUNT_DETAILS from the env
codingkarthik Jul 31, 2024
1643bdc
track cmd/gcp.go
codingkarthik Jul 31, 2024
b499d2c
update the github action
codingkarthik Jul 31, 2024
a2b554b
fix typo
codingkarthik Jul 31, 2024
9cb3e14
pass connector-publication-key as argument
codingkarthik Jul 31, 2024
41c35db
use env vars for sensitive stuff
codingkarthik Jul 31, 2024
6e759aa
no-op refactor
codingkarthik Jul 31, 2024
ec62c9f
no-op refactor
codingkarthik Aug 1, 2024
d5111d7
add a success log on completion
codingkarthik Aug 1, 2024
a9c0d3a
no-op refactors
codingkarthik Aug 1, 2024
c62a255
no-op change
codingkarthik Aug 2, 2024
c7f1891
remove stray comment
codingkarthik Aug 2, 2024
55784d2
update the registry-updates GH workflow
codingkarthik Aug 2, 2024
ee69f66
run registry-updates only against the main branch
codingkarthik Aug 2, 2024
372fdfb
Merge branch 'main' into kc/test-registry-automation
codingkarthik Aug 2, 2024
fa58cc5
restore stuff from main
codingkarthik Aug 2, 2024
9ede410
Add WIP RFC about the connector automation
codingkarthik Aug 2, 2024
f1966e8
update RFC
codingkarthik Aug 5, 2024
65191b5
rename legacy `package.json` file
codingkarthik Aug 5, 2024
4a881c2
remove copyright template
codingkarthik Aug 5, 2024
479f0c5
remove Copyright templates
codingkarthik Aug 5, 2024
128a2a0
address review comments
codingkarthik Aug 6, 2024
12ec043
update comment
codingkarthik Aug 6, 2024
5d5c7a8
make changes according to the new registry folder structure
codingkarthik Aug 7, 2024
c8cc3ee
add connector namespace
codingkarthik Aug 12, 2024
a0ba7fc
Manually set `PreBuiltDockerImage as package version type
codingkarthik Aug 12, 2024
baab34d
Merge branch 'main' into kc/test-registry-automation
codingkarthik Aug 12, 2024
bfd4909
add test connector-packages back
codingkarthik Aug 12, 2024
e47e759
Add relevant links in the RFCs
codingkarthik Aug 12, 2024
7af83a0
update the RFC
codingkarthik Aug 12, 2024
25fa3fe
fix TODOs
codingkarthik Aug 12, 2024
d53f63f
minor changes
codingkarthik Aug 12, 2024
10141a0
add notes in the RFC
codingkarthik Aug 13, 2024
6377629
final cleanup
codingkarthik Aug 13, 2024
25c4816
final cleanup
codingkarthik Aug 13, 2024
b39851d
Merge branch 'main' into kc/test-registry-automation
codingkarthik Aug 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/registry-updates.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Update Hub DB from GH Registry

on:
pull_request:
branches:
- main
types: [opened, synchronize, reopened]
paths:
- registry/**

jobs:
update_registry_db:
runs-on: ubuntu-latest
environment: staging

steps:
- name: Checkout repository
uses: actions/checkout@v2
with:
fetch_depth: 1

- name: Get all connector version package changes
id: connector-version-changed-files
uses: tj-actions/changed-files@v44
with:
json: true
escape_json: false
files: |
registry/**


- name: Print out all the changed filse
env:
ADDED_FILES: ${{ steps.connector-version-changed-files.outputs.added_files }}
MODIFIED_FILES: ${{ steps.connector-version-changed-files.outputs.modified_files }}
DELETED_FILES: ${{ steps.connector-version-changed-files.outputs.deleted_files }}
run: |
echo "{\"added_files\": $ADDED_FILES, \"modified_files\": $MODIFIED_FILES, \"deleted_files\": $DELETED_FILES}" > changed_files.json

- name: List changed files
id: list_files
run: |
cat changed_files.json

- name: Setup Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x

- name: Run registry automation program
env:
CHANGED_FILES_PATH: "changed_files.json"
PUBLICATION_ENV: "staging"
CONNECTOR_REGISTRY_GQL_URL: ${{ secrets.CONNECTOR_REGISTRY_GQL_URL }}
GCP_BUCKET_NAME: dev-connector-platform-registry
GCP_SERVICE_ACCOUNT_DETAILS: ${{ secrets.GCP_SERVICE_ACCOUNT_DETAILS }}
CONNECTOR_PUBLICATION_KEY: ${{ secrets.CONNECTOR_PUBLICATION_KEY }}
run: |
mv changed_files.json registry-automation/changed_files.json
cd registry-automation
go run main.go ci
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@

# testing
/tmp/empty
registry-automation/extracted_tgz
201 changes: 0 additions & 201 deletions LICENSE
codingkarthik marked this conversation as resolved.
Show resolved Hide resolved

This file was deleted.

1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,4 @@ implementation][NDC reference].

[NDC specification]: http://hasura.github.io/ndc-spec/
[NDC reference]: https://github.com/hasura/ndc-spec/tree/main/ndc-reference

Loading
Loading