- 2024-10-11/dmichaels
- Updated dcicutils version (8.16.1) for vulnerabilities. Updated cryptography version (43.0.1) for vulnerabilities.
Minor changes to the foursight_core/scripts/local_check_execution.py script. - Fix for --stage prod where we need to set os.environ["chalice_stage"]
so that the bucket is crected correctly (e.g. foursight-prod-mastertest).
- Print check results bucket for user FYI.
- Make it guess the AWS credentials name from AWS_PROFILE if no --env given.
- Fix to foursight_core/scripts/local_check_execution.py for newer SSO-based AWS credentials, where AWS_ACCESS_KEY_ID/etc environment variables are not set.
- Support for Python 3.12.
- Added ('[email protected] ', 'access_key_admin') to checks/access_key_expiration_detection.py.
- Fixed access_key_expiration_detection.py to get expiration time from data; not 90 days plus create date.
- Fix RedisSessionToken creation in react_api_base.ReactApiBase.react_authentication_callback to pass email which was missing (for some reaseon - previous oversight - was working only because ENCODED_REDIS_SERVER was not being defined in AWS Secrets Manager, e.g. for FoursightDevelopmentApplicationConfiguration)
- 2024-04-25/dmichaels
- Minor updates to scripts.local_check_execution to sanity check ElasticSearch connection.
- Update dcicutils version in pyproject.toml to specify an initial caret which was somehow missing.
- Other miscellaneous version updates in pyproject.toml; and ran poetry update.
- Fix in local-check-execution script causing script crash.
- 2024-03-03/dmichaels
- Added action_auto, action_manual, action_disable check decorator boolean or callable-returning-boolean options to force any associated action, after the check run, to run automatically, or to force it to not run automatically but to allow it to be run manually, or to not allow it to be run at allow, respectively. These are mutually exclusive; if more than one is set, then the first one to resolve to True, in reverse order (i.e. in order from disable, to manual, to auto) will be respected. NOTE: These work by setting the allow_check and prevent_check properties of the check result, and these new options will OVERRIDE these values which might have been expliclity set within the check code itself. NO CHANGES MADE TO EXISTING FUNCTIONALITY -> IF this new feature is NOT used.
- Set original_allow_action in app_utils.process_view_result to alleviate confusion from setting allow_action back to False in some cases to prevent re-running (TODO: revisit this at some point).
- Changes to the local-check-execute script.
- Fix to checks.test_checks.add_random_test_nums.
- UI fixed WRT preferred env name.
- Added update of a gitinfo.json file in GitHub Actions (.github/workflows/main-publish.yml).
- Fix to the (API for the) reindex page to NOT match the "initial" deploy task.
- Fix to the (API for the) redeploy and reindex pages for to straighten out blue/green and staging/data for clusters/services vs task definitions.
- Added indication to redeploy/reindex pages if in blue/green mirror stage.
- Added portal branch/commit to redeploy page.
- Minor UI fixes to the Ingestion page.
- Changed to NOT get Auth0 info from Portal; see foursight_core/react/api/auth0_config.py/PULL_AUTH0_INFO_FROM_PORTAL.
- New tasks page (initially to kick off Portal reindex and redeploy).
- Update poetry to 1.4.2.
- Update to Python 3.11.
- Minor UI updates.
2023-08
Support local-check-runner utility.
Minor change to respect REDIS_HOST_LOCAL environment variable (for local dev/testing), as well as allowing override of Auth0 client/secret (AUTH0_CLIENT_LOCAL, AUTH0_SECRET_LOCAL).
Miscellaneous changes to get foursight-smaht working properly. * Support to get consorita/submission_centers, as well as for
awards/labs for foursight-fourfront (previous oversight).
- Fixed up users pages.
Added UI warning bar about inability to connect to ElasticSearch.
A few minor UI tweaks; one WRT showing ff_link (like pre-React version) for check results.
2023-06-20
Changes to the access key check; making sure the action does not run every single day. This the primary/necessary change for this release; required since 4.3.0 where the access key check itself was fixed to work; without this new access keys would be created daily.
Replaced calls to boto3.client/resource("sqs"/"s3") to boto_sqs/s3_client/resource; this in preparation to allow using localstack to run SQS and S3 locally for testing; to really do this we need similar changes in dcicutils.
Miscellaneous minor UI improvements, including: * Allow viewing of list of secrets and values (obfuscated if senstive) in Infrastucture page. * Allow accounts file to be uploaded; this now lives in, for example:
s3://cgap-kmp-main-application-cgap-supertest-system/known_accounts No longer need to encrypt this file as it resides in a protected area in S3, i.e. the same place as the Portal access keys files (e.g. access_key_foursight).
- New info and convenience links to associated AWS resources on accounts page.
- Allow specifying UUID when creating a new user (C4-1050).
- Started adding ECS info to Infrastructure page.
- Fix to checks.access_key_expiration_detection.refresh_access_keys bug (key exception) which was preventing the portal access key from being updated in S3 (e.g. the keys access_key_admin, access_key_foursight, and access_key_tibanna in bucket gap-msa-main-application-cgap-msa-system).
- Minor UI updates related to: - Invalid/expired portal access key. - Checks search.
- Minor UI fixes for display of status text for checks/actions.
- Added UI warning for registered action functions with no associated check.
- Minor fix to not crash of Redis is enabled, i.e the REDIS_HOST environment variable is set (e.g. to redis://localhost:6379) but Redis is not actually running.
- Display Redis info on INFO page.
- Fix for check arguments not being converted (from string) to int/float/etc as appropriate in the React version only (was not calling query_params_to_literals).
- Support for catching SSL certificate errors for Portal.
- Using new dcicutils.scripts.publish_to_pypi for publish.
- Using new dcicutils.function_cache decorator for API function caching.
- Support for Redis Sessions if a server is configured from the GAC
- Some minor code changes based on (prematurely merged) PR-40.
- Somehow this verison got tagged/pushed prematurely.
- Support for AWS Cognito (to POSSIBLY eventually supplant Auth0 for login).
- Support for viewing and editing user projects and institutions (C4-945).
- Support for viewing info about VPCs, security groups, subnets, et cetera (C4-961).
- Support for viewing info (outputs, parameters, resources, templates) about stacks.
- Support for viewing checks by schedule.
- Support for simple checks search.
- Support for simple users search.
- New upper-left navigation dropdown menu.
- Changed all tooltips to use react-tooltip fixing issue with occlusion of adjacent elements.
- Support in React version for running actions and viewing their history.
- Fixed bug which prevented check with dependency from running because incorrectly looking up in S3 for the dependent check result; fix in run_check_runner in app_utils.py (see collect_run_info calls).
- Changes related to a new experimental /accounts page in Foursight which can show summary Foursight and Portal info from other AWS accounts. To take advantage of it there is an accounts.json file in the chalicelib_cgap or chalicelib_fourfront directory, for Foursight-CGAP and foursight-Fourfront, respectively, which contains a simple list of Foursight URLs for other AWS accounts. If this file is not present no harm. This file has been manually encrypted, since it contains internal URLs, using the ENCODED_AUTH0_SECRET value in the Global Application Configuration (GAC) in the AWS Secrets Manager. There are convenience poetry scripts, to encrypt and/or decrypt this file locally: encrypt-accounts-file, decrypt-accounts-file. Change to both the API and UI have been made for this.
- Moved lookup of check_setup.json (and accounts.json) to here, from foursight-cgap/chalicelib_cgap/app_utils.py and foursight/chalicelib_cgap/app_utils.py.
- Fix for C4-949: Show full_output as JSON (YAML actually) correctly in check run outputs.
- Disable user edit in readonly mode.
- Show indication in UI of how user logged in (i.e. Google or GitHub authentication).
- PEP8/PyCharm warning cleanup.
- Miscellaneous UI cleanup; very gradually more componentization and more consistent CSS.
- ElasticSearch 7 compatibility
- doc_types removed
- Add some common checks applicable everywhere
- Access Key Expiration Detection
- ECS Deployment Recovery Mechanism
- ES Snapshot Recovery Mechanism (not that usable)
- Datastore status check
- Scaling checks for RDS, ES
- Reintroduce tests for a large chunk of core functionality
- Split tests into "not integrated" and "integrated" runs as mock fixtures seem to "infect" other tests that do not use them
- React-ifying Foursight. * Many changes. * React UI code in react directory. * React API code in foursight_core/react directory. * Moved all foursight-cgap and foursight Chalice routes to here. * Old Foursight still works side-by-side with React version.
- Change some print statements to logger.debug in check_utils.py
- Spruced up Foursight UI a bit.
- New header/footer.
- Different looks for Foursight-CGAP (blue header) and Foursight-Fourfront (green header).
- More relevant info in header (login email, environment, stage).
- New /info and /users page.
- New /users and /users/{email} page.
- New dropdown to change environments.
- New logout link.
- New specific error if login fails due to no user record for environment.
- Repair changelong for 1.0.1
(There was also a need for this version to be higher than some beta versions that are in play.)
- Fix the way check lookup works
- Repair changelong for 1.0.1
(There was also a need for this version to be higher than some beta versions that are in play.)
- Fix the way check lookup works
- IDENTITY-ized Foursight; i.e. get secrets and other configuration data from the global application configuration (GAC) rather than having them encoded in the environment via the CloudFormation template (for the lambdas). C4-826.
- Added STACK_NAME (in addition to IDENTITY introduced in 0.7.4.2b0) to the environment variables required (via the Foursight CloudFormation template) to get a foothold for other Foursight info, e.g. to get the CHECK_RUNNER AWS lambda function name. See AppUtilsCore.apply_identity_globally in app_utils.py and Deploy.build_config in deploy.py. C4-826.
- Assume RDS_NAME is now in GAC (via 4dn-cloud-infra 3.4.0).
- Moved apply_identity_globally to its own identity.py module and call it statically from app_utils.py/AppUtilsCore class. C4-826.
- Merged in Kent's changes from PR-27 (to environment.py and check_utils.py WRT environment names).
- Merged in more of Kent's changes from PR-27 (to environment.py and check_utils.py WRT environment names).
- Added better error message for NoSuchKey for S3 bucket key access (s3_connection.py/get_object).
- Updated dcicutils to ^4.0.2.
Fix Environment.is_valid_environment_name to return true of various environments.
Fix a bug in Environment.get_environment_and_bucket_info
Relocked pyproject.toml from scratch. No other changes.
Versions 0.7.1 and 0.7.2 offer no change from 0.7.0 except version. Just trying to debug a problem at pypi.
PR #22: Miscellaneous changes in support of using new env_utils
- This tries to make use of the support in a recent utils beta to get a foothold on the foursight environment in a more abstract and configurable way.
PR #23: Mostly PEP8
- Address many PEP8 issues.
- Include
flake8among dev dependencies. - Add
make lintto runflake8. - Bring
CHANGELOG.rstup to date.
PR #21: Python 3.7 support (C4-765)
- Adjusts python requirement to permit Python 3.7, but still allow 3.6.1 and above. No known incompatibilities.
PR #20: Support Encrypted Buckets
- Small changes needed for encrypted buckets
(Records are uncertain here.)
PR #19: Repair delete_results
- Fix for problem where``delete_results`` had inconsistent return type,
causing
foursight-cgapto crash in the scenario where no checks are to be cleaned. With this change, it returns a tuple as the docstring says.
PR #18: Enable RDS Snapshots (1/3)
- Needs update
PR #17: Changes to remove variable imports from env_utils (C4-700)
- Needs update
PR #16: Remove dev_secret
There was no version 0.4.0.
PR #15: Update for dcicutils 2.0
PR #14: Add publishing support
PR #13: Fix C4-691 and C4-692 regarding information passing into foursight-core building operations
PR #9: foursight-core: chalice package support C4-554 (1/3)
Compatible/transitional support for:
- Fix for foursight-core Deploy.build_config_and_package should take global_env_bucket as an argument (C4-691): Allow environment variable (either one, checking for consistency if both are set) or an argument. If the argument is passed, it takes precedence even if not consistent with environment variables.
- Fix for foursight-core Deploy.build_config_and_package should not need an 'args' arg: Allow any of four new named arguments to override
the various parts of
argsthat might get used. So passingmerge_template=causes that value to be used in lieu ofargs.merge_template, andoutput_file=gets used in lieu ofargs.output_file, andstage=gets used instead ofargs.stage, andtrial=gets used in place ofargs.trial.
PR #12: Repair Auth0
PR #11: remove fuzzywuzzy dependency
- Needs more info
PR #10: Update buckets.py
PR #8: Collect run info
PR #6: delete check_runs_without_output function wfr_utils.py
PR #7: Fix visibility timeout
- SQS visibility timeout was set to 5 mins but should be 15 mins to reflect the updated lambda timeout.
There was no version 0.1.5
PR #5: fix for bug AppUtils object has no attribute get_schedule_names
PR #4: Core3
PR #3: Add GA Workflows
PR #2: Core2
- minor fixes
PR #1: Core2