Better channel restoration for local dev with content proxying

[bjester]

Summary

• Refactors restore_channel management command to be more robust and useful for restoring channels for local development:
  • No longer downloads content by default
  • Properly restores exercises if API token is provided
  • Supports making the channel public and publishing it after restoration
  • Better progress and logging of restoration
• Adds nginx to services started through Makefile make dcservicesup for proxying both devserver and content in development
• Adds composite and read-only storage pass through for development like non-production server environments
• Folds presigned URL handling for uploads into storage structure to eliminate duck typing

References

Supports ongoing improvements to curriculum automation (search recommendations)

Reviewer guidance

1. Restoring a channel: python contentcuration/manage.py restore_channel --editor [email protected] --source-url https://studio.learningequality.org --public --publish --token <your_studio_token> <channel_id> (the following is a small channel: d0ef6f71e4fe4e54bb87d7dab5eeaae2)
2. Start development server make run-services then pnpm run devserver
3. Log into local studio
4. Open the restored channel
5. Observe you can see the PDF and video resources without having downloaded them (proxying through nginx to the cloud)
6. Observe the channel exercises are properly restored
7. Observe new 'Download file' feature now does not simply open the file in a new tab

Notes

I simply deleted the existing tests for the channel restoration. Since it's a development only command, the tests usefulness are questionable, but I'm willing to add tests later if it's desired.

[akolson]

Is this a continuation from the previous line?

[bjester]

Yes, I believe an indent is the convention for parameter docstrings that wrap

[akolson]

What's the reason for moving to port 8081 again?

[bjester]

We're allocating port 8080 to nginx, so it can proxy through to the devserver and minio. Therefore the devserver needs to be on a different port, and this changes it to 8081. From the developer's perspective, nothing changes, because they still load the the 8080 URL, but in reality they're hitting nginx first instead of the raw python devserver.

[akolson]

From the developer's perspective, nothing changes, because they still load the the 8080 URL, but in reality they're hitting nginx first instead of the raw python devserver.

@bjester I did try to load the server on 8080 but was unable, I was successful with 8081 though.

[akolson]

The refactor make sense to me. I was able to restore 2 channels successfully (QA channel and the small channel recommended on the pr)--all seemed to work flawlessly including the restoration of exercises that were problematic with the previous script. Thanks @bjester

However, to access the server, you need to use port 8081 instead (as 8080 failed for me). I'm not sure if this in the intended behavior?

[bjester]

The refactor make sense to me. I was able to restore 2 channels successfully (QA channel and the small channel recommended on the pr)--all seemed to work flawlessly including the restoration of exercises that were problematic with the previous script. Thanks @bjester

However, to access the server, you need to use port 8081 instead (as 8080 failed for me). I'm not sure if this in the intended behavior?

No, this doesn't sound like intended behavior. It sounds like nginx wasn't running. Which commands did you use? Do you see any output about the nginx container crashing?