v5.0.0

Release v5.0.0

Upgrading

This is a major release and it not compatible with any previous versions.
To use this version you must start a new TRE deployment.

Changes

• Complete rewrite of code in Python using IAC and configuration management tools Pulumi and Ansible

What's Changed

• Release v4.0.1 candidate by @jemrobinson in #1324
• Proof-of-concept migration to Pulumi for deployment by @jemrobinson in #1316
• Release v4.0.2 candidate by @jemrobinson in #1353
• Release v4.0.3 candidate by @jemrobinson in #1365
• Add instructions for installing documentation build dependencies by @JimMadge in #1370
• Update docs with how to resize VMs by @edwardchalstrey1 in #1367
• Update Badges by @JimMadge in #1371
• Update Powershell module requirements by @craddm in #1368
• Allow -UseDeviceAuthentication switch in Deploy_SHM.ps1 by @craddm in #1378
• Prevent removal of backup data during dry run by @JimMadge in #1383
• Pulumi: Fix user list retrieval by @craddm in #1386
• Policy for software package requests by @jemrobinson in #1387
• Add firewall to Pulumi by @jemrobinson in #1375
• Add arrow CRAN package to Tier 3 allowlist by @craddm in #1391
• ⬆️ Update caching in allowlists workflow by @jemrobinson in #1395
• Update user management guide to explain adding users to security group and changing a phone number by @edwardchalstrey1 in #1389
• Add Python type-hinting throughout Pulumi codebase by @jemrobinson in #1390
• Add instructions for GPU VM resizing by @edwardchalstrey1 in #1399
• Simplify Pulumi secret handling by @jemrobinson in #1400
• Add separate docs section GPU VMs and specify NVIDIA required by @edwardchalstrey1 in #1406
• Add Linux update server proxy by @jemrobinson in #1404
• Remove reference to unused System Administrators Security Group by @edwardchalstrey1 in #1407
• Add automated updates to Pulumi by @jemrobinson in #1412
• Refactor SRD creation by @jemrobinson in #1416
• Add SHM bastion by @jemrobinson in #1417
• Fix allowlist generation by @jemrobinson in #1422
• Update SRD image by @jemrobinson in #1421
• Fix incorrect logic around automated PR creation by @jemrobinson in #1426
• Update PyPI and CRAN allow lists by @github-actions in #1425
• Add new servicebus endpoints for self-service password reset by @edwardchalstrey1 in #1423
• Update PyPI and CRAN allow lists by @github-actions in #1428
• Update PyPI and CRAN allow lists by @github-actions in #1429
• Remove egress steps not carried out by System Manager by @edwardchalstrey1 in #1434
• Update SRE user troubleshooting by @edwardchalstrey1 in #1435
• Update SRD package versions by @github-actions in #1433
• Update PyPI and CRAN allow lists by @github-actions in #1437
• Update SRD package versions by @github-actions in #1440
• Add RPostgreSQL to t3 extra cran allowlist by @edwardchalstrey1 in #1441
• Revert "Add RPostgreSQL to t3 extra cran allowlist" by @JimMadge in #1442
• Better package name matching for Nexus by @craddm in #1447
• Update PyPI and CRAN allow lists by @github-actions in #1454
• Update PyPI and CRAN allow lists by @github-actions in #1456
• Update SRD package versions by @github-actions in #1460
• Update VM resizing note to suggest stopping the VM before increasing the quota by @edwardchalstrey1 in #1408
• Add data preparation guidance (including data integrity) by @JimMadge in #1459
• Migrate docs to readthedocs.io by @JimMadge in #1453
• Create users with no password expiry on AD by @craddm in #1461
• Modify location of requirements.txt in Dockerfile by @craddm in #1464
• Merge documentation changes into release branch by @JimMadge in #1468
• cherrypick devcontainer fix to release branch by @JimMadge in #1469
• Update servicebus endpoints used for self-service password reset by @jemrobinson in #1466
• Correct path to Scriberia cartoon in README.md by @JimMadge in #1475
• Replace deprecated Set-AzDiagnosticSetting by @jemrobinson in #1470
• Update PyPI and CRAN allow lists by @github-actions in #1477
• Correct link on citation badge by @JimMadge in #1474
• Add CODEOWNERS for docs by @jemrobinson in #1478
• Update documentation dependencies by @JimMadge in #1476
• Enable pdf and html downloads on readthedocs by @JimMadge in #1462
• Update SRD package versions by @github-actions in #1482
• Updating SSL certificate doc + gitignore change + undo duplication of docs building by @edwardchalstrey1 in #1432
• Mount data and user directories in SRD by @jemrobinson in #1480
• Change servicebus firewall rule by @craddm in #1485
• Folder typo for SHM deployment by @edwardchalstrey1 in #1488
• Update SRD package versions by @github-actions in #1489
• Force az login before reading Pulumi encryption key by @jemrobinson in #1490
• Clarify PR template by @jemrobinson in #1491
• Offline linkcheck by @JimMadge in #1486
• Pulumi: Add Git and Markdown servers by @jemrobinson in #1492
• Fixing the build warnings for documentation by @craddm in #1483
• Add Nexus repositories by @jemrobinson in #1499
• Pin container images by @JimMadge in #1501
• Automate user synchronisation by @jemrobinson in #1500
• Switch CLI interface to Typer by @jemrobinson in #1502
• Refactor config files by @jemrobinson in #1510
• Add portal.azure.com to lychee ignore list by @JimMadge in #1520
• Bump certifi from 2023.5.7 to 2023.7.22 in /docs by @dependabot in https://git...

Release v5.0.0rc2

Release v5.0.0rc2

This release is not ready for production usage.

Known Issues

• ClamAV not configured
• Unstable container service IP addresses
• Lacking Nvidia utils

What's Changed

• Use pip-compile for package resolution by @jemrobinson in #1514
• Add pip-tools to NON_IMPORTABLE_PACKAGES by @edwardchalstrey1 in #1537
• Add May 2023 DSG to versioning by @jemrobinson in #1545
• Release v4.1.0 cloud init changes by @edwardchalstrey1 in #1548
• Update SRD package versions by @github-actions in #1578
• Update PyPI and CRAN allow lists by @github-actions in #1579
• Fix deployment issues with MSSQL and PyPi mirrors by @craddm in #1582
• Update PyPI and CRAN allow lists by @github-actions in #1588
• Update SRD package versions by @github-actions in #1587
• Updates for Release v4.1.0 by @craddm in #1590
• Release v4.1.0 by @craddm in #1586
• Remove CoCalc by @craddm in #1554
• Merge 'latest' into 'develop' by @craddm in #1593
• Add script to automate account deletion by @edwardchalstrey1 in #1508
• Add @craddm to CODEOWNERS by @jemrobinson in #1594
• Update PyPI and CRAN allow lists by @github-actions in #1595
• Remove pulumi testing files from develop branch by @craddm in #1597
• Update PyPI and CRAN allow lists by @github-actions in #1601
• Update SRD package versions by @github-actions in #1616
• Update SRD package versions by @github-actions in #1622
• Bump urllib3 from 2.0.2 to 2.0.6 in /docs by @dependabot in #1625
• Improve Pulumi error messages by @craddm in #1624
• Update PyPI and CRAN allow lists by @github-actions in #1627
• Update PyPI and CRAN allow lists by @github-actions in #1631
• Update SRD package versions by @github-actions in #1630
• Improve Python documentation by @jemrobinson in #1635
• Use Pulumi random provider by @jemrobinson in #1629
• Pulumi: Fix selectors not updating by @JimMadge in #1621
• Bump urllib3 from 2.0.6 to 2.0.7 in /docs by @dependabot in #1647
• Remove hyphens from SHM and SRE names by @craddm in #1650
• Update PyPI and CRAN allow lists by @github-actions in #1646
• Update SRD package versions by @github-actions in #1652
• Pulumi: Improve login flow by @JimMadge in #1617
• Update PyPI and CRAN allow lists by @github-actions in #1654
• Add all contributors table and instructions for how to update by @edwardchalstrey1 in #1649
• Update PyPI and CRAN allow lists by @github-actions in #1656
• Update PyPI and CRAN allow lists by @github-actions in #1668
• Update SRD package versions by @github-actions in #1669
• Update devcontainer configuration by @craddm in #1662
• Update outdated parameters that cause breaking change warnings by @craddm in #1663
• Change default lun from lun1 to lun0 by @craddm in #1667
• Add context command by @JimMadge in #1655
• Pulumi: Update dependencies, enable pinning by @JimMadge in #1660
• Remove unneeded opening bracket in SRE network configuration script by @craddm in #1670
• Update PyPI and CRAN allow lists by @github-actions in #1671
• Use memory for the /tmp directory by @craddm in #1672
• Factor out storage creation from SHM scripts by @craddm in #1673
• Add missing import for logging module by @JimMadge in #1681
• Update PyPI and CRAN allow lists by @github-actions in #1682
• Update help text for Powershell command shmId andsreId arguments by @craddm in #1683
• Update contributors by @JimMadge in #1684
• Document removal of persistent SRE storage accounts by @craddm in #1685
• docs: update @helendduncan as a contributor by @JimMadge in #1686
• Update PyPI and CRAN allow lists by @github-actions in #1688
• Update SRD package versions by @github-actions in #1692
• Update PyPI and CRAN allow lists by @github-actions in #1693
• Update PyPI and CRAN allow lists by @github-actions in #1694
• Update DBeaver drivers using Github workflow by @craddm in #1696
• Update SRD package versions by @github-actions in #1698
• Bump jinja2 from 3.1.2 to 3.1.3 in /docs by @dependabot in #1700
• Update SRD package versions by @github-actions in #1701
• Update PyPI and CRAN allow lists by @github-actions in #1702
• Update PyPI and CRAN allow lists by @github-actions in #1703
• Handle no selected context by @JimMadge in #1691
• Add basic config commands by @JimMadge in #1674
• Fixing DBeaver driver issues on T2+ SREs by @craddm in #1704
• Use Pydantic for validation and serialisation by @JimMadge in #1661
• Improve handling of spaces in file paths by @craddm in #1705
• Update PyPI and CRAN allow lists by @github-actions in #1706
• Create pulumi container by @jemrobinson in #1711
• Fix private link scope by @jemrobinson in #1713
• Improve handling of SRE names by @JimMadge in #1699
• Apply changes from updated black version by @jemrobinson in #1718
• Bump black version by @JimMadge in #1719
• Fix some issues with context handling at deployment time by @jemrobinson in #1716
• Update SRD package versions by @github-actions in #1723
• Correct file path for clamonacc service by @craddm in #1725
• Add additional multiple data provider guidance to docs by @craddm in #1707
• Update SRD package versions by @github-actions in #1727
• Fix Pos...

Release 4.2.2 (2024-07-15)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.2.x SHM and want to upgrade to 4.2.2, please follow the steps below:

For the SHM:

1. Add a docker section to your SHM config with a username and personal access token (following the SHM deployment instructions)
2. Re-run Setup_SHM_Networking.ps1 -shmId {shm} from deployment/safe_haven_management/setup

For any SRE that you deployed using an earlier 4.2.x version:

1. Delete the GUACAMOLE-SRE-{sreId} VM and associated resources from the
   RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP resource group
2. Re-run the deployment script Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before} from deployment/secure_research_environment/setup

Known issues

• As for 4.2.0, 4.2.1

Bug Fixes

• Workaround for an issue where Let's Encrypt refused to provide certificates for uppercase FQDNs #1938
• Fix for change in Azure supported public IP address SKU for VPNs, which prevented deployment of the virtual network gateway for accessing domain controllers #1947
• Require supply of Docker Hub credentials to work round change in Docker download rate limits #1994
• Update approved IP address list for Ubuntu apt repositories
• Update to backup policy rules for Blob storage #1988

Full Changelog: v4.2.1...v4.2.2

Release v4.2.1 (2024-05-31)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.2.0 SHM and want to upgrade to 4.2.1, please follow the steps below:

1. Delete the GUACAMOLE-SRE-{sreId} VM and associated resources from the RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP resource group
2. Re-run the deployment script Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before} from deployment/secure_research_environment/setup

Known issues

• As for 4.2.0

Bug Fixes

• Update Guacamole to 1.5.5 to avoid this known bug

Full Changelog: v4.2.0...v4.2.1

Release 4.2.0 (2024-03-28)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.1.0 SHM and want to upgrade to 4.2.0, please follow the steps below:

1. Run Setup_SHM_Firewall.ps1 -shmId {shmid}
2. Run Setup_SHM_Networking.ps1 -shmId {shmid}
3. Delete LINUX-UPDATES-SHM-{shmid} VM and associated resources from the RG_SHM_{shmid}_MONITORING resource group
4. Delete RG_SHM_{shmid}_PACKAGE_REPOSITORIES resource group and all resources
5. Run Setup_SHM_Update_Servers.ps1 -shmId {shmid} (Note that this needs to happen before any further resources are deployed, since any further Linux resources will need access to the Linux update proxy).
6. Run Setup_SHM_Package_Repositories -shmId {shmid}
7. Run Setup_SHM_Monitoring.ps1 -shmId {shmid}

Known issues

• Jupyter notebook launched from GUI menu could not launch Python kernel, so it has been removed from the menu 0657647

New Features

• Remove Microsoft Remote Desktop support: #1535
• Remove CoCalc: #1554
• Install dev dependencies in container: #1747
• Add script to renew NFS share Stored Access Policies: #1739
• Add script to automate account deletion: #1508
• Factored out storage creation from SHM scripts #1673
• SRD image updated, with latest Python versions available f3e890a

Bug Fixes

• Update DBeaver drivers using Github workflow: #1696
• Fixing DBeaver driver issues on T2+ SREs: #1704
• Improve handling of spaces in file paths: #1705
• Correct file path for Clam OnAccess scanning service: #1725
• Fix PostgreSQL permissions and data schema, and relevant docs: #1708
• Update outdated parameters that cause breaking change warnings: #1663
• Change default lun from lun1 to lun0: #1667
• Increase apt proxy server disk to 64 Gb: #1726
• Remove omsagent from VM build image: #1732
• Remove hyphens from SHM and SRE names in #1650
• Update devcontainer configuration in #1662
• Use memory for the /tmp directory in #1672
• Remove unneeded opening bracket in SRE network configuration script #1670
• Add missing import for logging module #1681
• Fix cloud-init log parser using old name for event 58a85bc
• Detect and remove omsagent installed on SRD image before generalization e168b05

Security Fixes

• Update software on Guacamole and Nginx to latest versions: #1741
• Update Nexus proxy server for T2/T3 package access: in #1744
• Update CodiMD server version: #1743
• Improve hardcoded domains and IP addresses: #1745
• Prevent Nginx version information from appearing in http headers

Documentation updates

• Add guidance on resizing NFS shares: #1749
• Update documents to reflect change to Microsoft Entra ID: #1665
• Update deprecation warning for MS RDS: #1542
• Add explanation of how to change allowed inbound IP addresses: #1484
• Add all contributors table and instructions for how to update: #1649
• Update contributors: #1684
• Document removal of persistent SRE storage accounts: #1685
• docs: update contributors: #1686
• Add additional multiple data provider guidance to docs: #1707
• Add links to guides for terminal, Xfce, and Guacamole: #1737
• Update help text for Powershell command shmId andsreId arguments #1683

Full Changelog: v4.1.0...v4.2.0

Release v5.0.0-rc.1 (2023-09-27)

First version of migration to Python using Pulumi. Penetration tested in September 2023.

Known Issues

This release is not ready for production usage.

Release 4.1.0 (2023-09-06)

⚠️ Update Requires Manual Intervention ⚠️

If you are using a 4.X.Y SHM and want to upgrade to 4.1.0, please follow the steps below:

• Run ./deployment/safe_haven_management/setup/Setup_SHM_Networking.ps1 -shmId <your SHM ID>
• Restart the virtual machine at RG_SHM_<SHM name>_MONITORING/LINUX-UPDATES-SHM-<SHM name> in the Azure portal

Known Issues

Only phone call authentication works for MS RDS. This provides no on-screen MFA Prompt.

New Features

• Allow device authentication in SHM deployment #1378
• Add arrow CRAN package to Tier 3 core list #1391
• Update Python in SRD images #1421

Bug Fixes

• Update Powershell module requirements: #1368
• Update supported Powershell version to 7.3.6
• Prevent removal of backup data during dry run: #1383
• Better package name matching for Nexus: #1447
• Update SRD image: #1421
• Add new servicebus endpoints for self-service password reset: #1423 and #1466
• Modify location of requirements.txt in Dockerfile: #1469
• Fixes of the SRD build related to python packages: #1514 and #1537
• Fix allowlist generation: #1422
• Update badges: #1371
• Update caching in allowlists workflow: #1395
• Fix incorrect logic around automated PR creation: #1426
• Update Ubuntu apt server addresses #1548
• Add docker.io to allowed-FQDNs #1548
• Change cloud-init files to automatically select appropriate disk partition #1548
• Fix MS-SQL database deployment #1580
• Fix PyPi Tier 3 mirror failures #1581

Security Fixes

• Fix non-allowed CRAN packages beginning with allowed name being installable: #1447
• Update to firewall rules: #1519

Documentation Updates

• Add instructions for installing documentation build dependencies: #1370
• Add instructions to resize VMs: #1367
• Update user management guide to explain adding users to security group and changing a phone number: #1389
• Add instructions for GPU VM resizing: #1399
• Add note on NVIDIA GPU support: #1406
• Remove reference to unused System Administrators Security Group: #1407
• Remove egress steps not carried out by System Manager: #1434
• Update SRE user troubleshooting: #1435
• Move from GitHub pages to ReadTheDocs #1468
• Add Policy for software package requests: #1387
• Add deprecation warning for MSRDS #1542
• Add warning that MSRDS does not work with the Microsoft Authentication app. #1589
• Add step for adding SSL certificate in step-by-step instructions for Guacamole #1590

Full Changelog: v4.0.3...release-v4.1.0

Release 4.0.3 (2023-01-27)

Bug fixes

• Update maximum allowed Powershell version
• Fix disk mounting issue when upgrading SRDs

Documentation updates

• Minor fixes

Release 4.0.2 (2023-01-05)

Bug fixes

• Add missing Powershell module imports
• Fix -Upgrade option when adding new SRD
• Fix tensorflow installation in SRD base image
• Register Microsoft.DataProtection on subscriptions that an SRE will be deployed into
• Support cross-subscription role assignments for backup
• Switch to correct subscription before deploying update automation
• Update Powershell version requirements to avoid upstream bug
• Update SRD package versions
• Use process-scope when retrieving Graph authorization tokens with Connect-MgGraph

Security fixes

• Remove unnecessary information from deployment logging

Documentation updates

• Add link to teardown docs to deployment page
• Add a VSCode .devcontainer for use in deployment
• Clarify that IP addresses are required in SRE config file
• Consolidate MFA setup description
• Update documentation build triggers to also run on latest

Release 4.0.1 (2022-10-24)

Bug fixes

• Add additional modules to requirements checker
• Add check for non-existing AzureAD security group
• Switch CI tests from Travis to GitHub Actions

Documentation updates

• Updated issue templates
• Fix documentation building