Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify export_opportunity_data_task to create ExtractMetadata database records #2794

Open
2 tasks
mikehgrantsgov opened this issue Nov 8, 2024 · 0 comments · May be fixed by #2998
Open
2 tasks

Modify export_opportunity_data_task to create ExtractMetadata database records #2794

mikehgrantsgov opened this issue Nov 8, 2024 · 0 comments · May be fixed by #2998
Labels
topic: backend Backend development tickets

Comments

@mikehgrantsgov
Copy link
Collaborator

mikehgrantsgov commented Nov 8, 2024

Related to 2454

Summary

2793 creates the API needed to send records back to the database. This ticket populates that table with results from the export_opportunity_data_task task.

Something like:

def run_task(self) -> None:

   .... existing code ....
   self.record_metadata("json", self.json_file)
   self.record_metadata("csv", self.csv_file)

   def record_metadata(self, extract_type: str, file_path: str) -> None:
    file_name = os.path.basename(file_path)
    file_size = os.path.getsize(file_path)
    metadata = ExtractMetadata(
        extract_type=extract_type,
        file_name=file_name,
        file_path=file_path,
        file_size=file_size
    )
    self.db_session.add(metadata)
    self.db_session.commit()

Note: The file path should be a public URL to the file for download. If we go a presigned URL, we will need a follow-up task to generate presigned URLs in the ExtractMetadata response.

Acceptance criteria

  • Opportunity export task updated to add db rows
  • Tests added to test_export_opportunity_data_task to make sure db records are being created.
@mikehgrantsgov mikehgrantsgov added the topic: backend Backend development tickets label Nov 8, 2024
mikehgrantsgov added a commit that referenced this issue Nov 18, 2024
…files (#2880)

## Summary
Fixes #2791

### Time to review: 10 mins

## Changes proposed
- Add a lookup table `LkExtractType`
- Add a `ExtractMetadata` model and associated table for future use by
the backend.

## Context for reviewers
This issue unblocks many other backend tickets from creating and storing
the necessary db extracts in S3.
See #2792, #2793, #2794

## Additional information
See generated migration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: backend Backend development tickets
Projects
1 participant