Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gh 621 bulk import #815

Closed
wants to merge 20 commits into from
Closed

Conversation

chb0github
Copy link
Contributor

@chb0github chb0github commented Oct 2, 2023

Ok, this is working, but I am working on an improvement.

You see, if I fetch all the repo data, regardless of type, then I just remap it all. But, it turns out only docker local is our problem child, and the dockerApiVersion is the only thing I can't know with a single API call. I'll post the new code that carves these out as an exception for "performance" reasons - only those repos will get the double fetch which, definitely will result in a lot less calls, but for a lot more code (see below).

to use this tool, you run:

./bulkimport.sh --repos --users -g -h "${host}" > import.tf
# out="$RANDOM-out"
#terraform plan -generate-config-out generated.tf -out "${out} -parallelism=10
#terraform apply -parallelism=10

Duplicate resource requests are discarded

chb0github and others added 4 commits September 25, 2023 17:22
next steps: handle the special case of docker local repos (we need to fetch the version number which requires an extra call)
test management of netrc (we don't want to mess up someones file)
exclude resources found in the local terraform state
Dockerfile Show resolved Hide resolved
sample.tf Outdated Show resolved Hide resolved
scripts/bulkimport.sh Outdated Show resolved Hide resolved
scripts/bulkimport.sh Outdated Show resolved Hide resolved
scripts/repodata.jq Outdated Show resolved Hide resolved
@alexhung alexhung linked an issue Oct 2, 2023 that may be closed by this pull request
@alexhung alexhung self-requested a review October 2, 2023 16:31
@alexhung alexhung added the enhancement New feature or request label Oct 2, 2023
* master:
  Update devfile.yaml
  trying out spaces
@alexhung alexhung added the no changelog No CHANGELOG.md update label Oct 2, 2023
@chb0github chb0github marked this pull request as ready for review October 2, 2023 23:35
@alexhung
Copy link
Member

alexhung commented Oct 2, 2023

@chb0github Looks like I was wrong and Docker v1 is still very much supported.

Screenshot 2023-10-02 at 4 35 28 PM

@chb0github
Copy link
Contributor Author

@alexhung - no, it doesn't actually try clicking v1 and hitting "submit"

@alexhung
Copy link
Member

alexhung commented Oct 2, 2023

@chb0github No problem creating a local Docker v1 repository

Screenshot 2023-10-02 at 4 38 40 PM

@chb0github
Copy link
Contributor Author

image

@chb0github
Copy link
Contributor Author

So, do you want me to add the legacy code for v1, or is this good to commit and get feedback

@alexhung
Copy link
Member

alexhung commented Oct 4, 2023

@chb0github So the Docker v1 issue seems to be a bug from Artifactory. The version you are using is 7.56.0, which is old. The local version I am running is 7.68.7 which support v1 again. Same with latest cloud version 7.69.4.

So we need to have Docker v1 support in the script.

…overed a bug in the netrc logic that needs fixing (removing for now).
…form-provider-artifactory into jfrogGH-621-bulk-import

* 'jfrogGH-621-bulk-import' of github.com:chb0github/terraform-provider-artifactory:
  fix store_artifacts_locally docs to reflect reality
  JFrog Pipelines - Add Artifactory version to CHANGELOG.md
  Update CHANGELOG
  Update module path from 'v8' to 'v9'
  Add planmodifier.RequiresReplace() to "key" attribute
  Update CHANGELOG
  Migrate 'artifactory_backup' resource to terraform-plugin-framework
  Update CHANGELOG
  Update validation for "project_environments" to allow empty list
  Update documentation
  JFrog Pipelines - Add Artifactory version to CHANGELOG.md
  Update CHANGELOG
  Add "force_conan_authentication" attribute to local, virtual, and federated repo resources and data sources
@chb0github
Copy link
Contributor Author

docker v1 support is added, and the script is updated

alexhung
alexhung previously approved these changes Oct 9, 2023
Copy link
Member

@alexhung alexhung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the Terraform cli installation in Dockerfile, all looks fine to me.

sample.tf Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
sample.tf Outdated Show resolved Hide resolved
@chb0github
Copy link
Contributor Author

I'm not particularly happy with the internal code formatting due to the need to output also with format to the import file, but using JQ was 6x faster than using the shell. I was shocked.

@chb0github
Copy link
Contributor Author

If there is some input in the future, please tag me

scripts/bulkimport.sh Outdated Show resolved Hide resolved
@chb0github
Copy link
Contributor Author

@danielmkn waiting on you, buddy

@danielmkn
Copy link
Collaborator

@chb0github, can you please add some documentation on that?

@chb0github
Copy link
Contributor Author

Where would you like me to put it? It was marked as no change log

Please confer with Alex. Also, the app is self documenting, per best practices

@chb0github
Copy link
Contributor Author

@danielmkn - where would you like this documentation? And, did you consult with @alexhung ?

@chb0github
Copy link
Contributor Author

Hey Guys, if we're not going to merge it in, then an explanation why would be great.

@alexhung
Copy link
Member

Note We will put this PR on hold for the immediate future. Factors that contribute to this decision include:

  • This utility at its current state requires additional effort to support more resource types as well as 'productization', e.g. testing, documentation, etc.
  • JFrog's team inability to support this tool for the time being. Our plan for Q4 2023 is already set and we can revisit when we plan Q1 2024.

In the meantime, this PR will remain open and available for others to copy and use in their own trials and experiments. We welcome any suggestions or feedback so when we are ready to work on this in the future, we will have a better idea on what our customers want.

@alexhung alexhung added the hold On hold label Oct 16, 2023
@chb0github
Copy link
Contributor Author

Ok, So, having thought about this more, a more terraform idiosyncratic approach to this would be to :

  1. Implement all artifactory resource types as data_sources
  2. then, combine with a file data source and output the same format being generated by my script to a file_source

Although, right now the data source for repositories is still pending merge, and there aren't data sources for the other resources.

@chb0github chb0github closed this Jan 28, 2024
@chb0github chb0github deleted the GH-621-bulk-import branch January 28, 2024 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request hold On hold no changelog No CHANGELOG.md update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mass import existing resources from JFrog into Terraform
3 participants