Replace hardcoded validation cases

[seallard]

Description

This PR replaces the hard coded case id:s in cg that should be ignored with a database field. Closes #2607.

TODO:

• Add revision with is_compressible
• Update logic to check the is_compressible field
• Write test to verify that cases marked as not compressible are ignored.
• Write script to set is_compressible to false for all validation cases. Store comments on case.
• Apply revision to stage and run script. Verify that it works.
• Merge and do the same for prod

Changed

• Use is_compressible field instead of hard coded cases to ignore

This version is a

• PATCH - when you make backwards compatible bug fixes or documentation/instructions

[seallard]

Script to mark validation cases in status db

from cg.store.api.core import Store
from cg.store.database import initialize_database

PROBLEMATIC_CASES = [
    ("causalmite", ""),
    ("deepcub", ""),
    ("expertalien", ""),
    ("fluenteagle", ""),
    ("grandkoi", ""),
    ("lovingmayfly", ""),
    ("loyalegret", ""),
    ("modernbee", ""),
    ("proudcollie", ""),
    ("richalien", ""),
    ("suremako", ""),
    ("wisestork", ""),
]

BALSAMIC_VALIDATION_CASES = [
    ("setamoeba", "BALSAMIC validation case tumor-only panel"),
    ("sweetelf", "BALSAMIC positive control tumor-only panel"),
    ("poeticghoul", "BALSAMIC positive control tumor-only panel"),
    ("equalbug", "UMI seracare validation case (AF 0.5%) tumor-normal panel"),
    ("stableraven", "UMI seracare validation case (AF 1%) tumor-normal panel"),
    ("uphippo", "UMI seracare validation case (AF 0.5%) tumor-only panel"),
    ("cleanfowl", "BALSAMIC validation case, HD829 reference for FLT3 Ascertation"),
    ("proudsquid", "BALSAMIC validation case, HD829 reference for FLT3 Ascertation"),
    ("modestjaguar", "BALSAMIC validation case, HD829 reference for FLT3 Ascertation"),
    ("dearmarmot", "BALSAMIC validation case, HD829 reference for FLT3 Ascertation"),
    ("holykid", "BALSAMIC validation case, HD829 reference for FLT3 Ascertation"),
    ("civilsole", "BALSAMIC validation case tumor-only wgs"),
    ("fleetjay", "BALSAMIC validation case tumor-normal wgs"),
    ("grandmarmot", "BALSAMIC validation case tumor-normal wgs"),
    ("unitedbeagle", "BALSAMIC validation case tumor-normal panel"),
    ("rightthrush", "BALSAMIC validation case tumor-only wes"),
    ("properpigeon", "BALSAMIC validation case tumor-normal wes"),
    ("eagerox", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("casualweasel", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("acetuna", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("suitedsnake", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("savinghorse", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("rightpup", "BALSAMIC validation case from cust087, tumor-only panel"),
    ("sureroughy", "BALSAMIC validation case from cust087, tumor-only panel"),
    (
        "notedshark",
        "BALSAMIC validation case from cust127, tumor-normal WGS (SV inversion positive control)",
    ),
    (
        "wholewhale",
        "BALSAMIC validation case from cust143, tumor-normal WGS (SV translocation positive control)",
    ),
    (
        "largeturtle",
        "BALSAMIC validation case from cust143, tumor-normal WGS (SV translocation positive control)",
    ),
    (
        "lightkodiak",
        "BALSAMIC validation case from cust143, tumor-normal WGS (SV inversion positive control)",
    ),
    (
        "wholecivet",
        "BALSAMIC validation case from cust110, tumor-normal WGS (SV inversion positive control)",
    ),
    (
        "upwardstork",
        "BALSAMIC validation case from cust110, tumor-normal WGS (SV deletion positive control)",
    ),
    (
        "suitedgrub",
        "BALSAMIC validation case from cust110, tumor-normal WGS (SV deletion positive control)",
    ),
]

FLUFFY_VALIDATION_CASES = [("simplesalmon", "Chromosome 13, 18, 21 Suspected")]

MIP_VALIDATION_CASES = [
    ("brightcaiman", "DNA rare disease positive control"),
    ("casualgannet", "DNA rare disease positive control"),
    ("civilkoala", "RNA rare disease positive control"),
    ("cleanshrimp", "DNA rare disease positive control"),
    ("drivenmolly", "RNA rare disease positive control"),
    ("easybeetle", "DNA rare disease positive control"),
    ("epicasp", "DNA rare disease positive control"),
    ("expertmole", "RNA rare disease positive control"),
    ("finequagga", "RNA rare disease positive control"),
    ("firstfawn", "DNA rare disease positive control"),
    ("gladthrush", "DNA rare disease positive control"),
    ("helpedfilly", "DNA rare disease positive control"),
    ("hotskink", "DNA rare disease positive control"),
    ("inferret", "DNA rare disease positive control"),
    ("intentcorgi", "DNA rare disease positive control"),
    ("intentmayfly", "DNA rare disease positive control"),
    ("justhusky", "DNA rare disease positive control"),
    ("kindcaiman", "DNA rare disease positive control"),
    ("lightprawn", "DNA rare disease positive control"),
    ("livingox", "DNA rare disease positive control"),
    ("newaphid", "RNA rare disease positive control"),
    ("nextjackal", "DNA rare disease positive control"),
    ("modernmule", "DNA rare disease positive control"),
    ("moralcattle", "RNA rare disease positive control"),
    ("onemite", "DNA rare disease positive control"),
    ("opencow", "DNA rare disease positive control"),
    ("proudcougar", "DNA rare disease positive control"),
    ("rightmacaw", "DNA rare disease positive control"),
    ("safeguinea", "DNA rare disease positive control"),
    ("sharpparrot", "RNA rare disease positive control"),
    ("sharppigeon", "DNA rare disease positive control"),
    ("sharpwhale", "DNA rare disease positive control"),
    ("stillant", "DNA rare disease positive control"),
    ("smoothboa", "RNA rare disease positive control"),
    ("strongbison", "DNA rare disease positive control"),
    ("tenderoriole", "DNA rare disease positive control"),
    ("topsrhino", "DNA rare disease positive control"),
    ("usablemarten", "DNA rare disease positive control"),
    ("vitalmouse", "DNA rare disease positive control"),
]

RNAFUSION_VALIDATION_CASES = [
    ("ableheron", ""),
    ("acecoyote", ""),
    ("ampleray", ""),
    ("bossmink", ""),
    ("cuddlyhen", ""),
    ("daringowl", ""),
    ("expertboar", ""),
    ("finerracer", ""),
    ("growndoe", ""),
    ("guidedfeline", ""),
    ("handyturtle", ""),
    ("hardygrouse", ""),
    ("holyrodent", ""),
    ("honestswine", ""),
    ("inlab", ""),
    ("justburro", ""),
    ("movedmule", ""),
    ("nearbyjoey", ""),
    ("oncrab", ""),
    ("politeglider", ""),
    ("rareosprey", ""),
    ("rightmoray", ""),
    ("stablemoray", ""),
    ("starnewt", ""),
    ("tendergoose", ""),
    ("truemole", ""),
    ("truepigeon", ""),
    ("valuedfowl", ""),
    ("vocallocust", ""),
    ("wantedsawfly", ""),
]

TAXPROFILER_VALIDATION_CASES = [("richurchin", "")]

OTHER_VALIDATION_CASES = [
    ("bigdrum", ""),
    ("busycolt", ""),
    ("daringpony", ""),
    ("frankhusky", ""),
    ("gamedeer", ""),
    ("hotviper", ""),
    ("keencalf", ""),
    ("keenviper", ""),
    ("luckyhog", ""),
    ("maturejay", "sars-cov-2 case"),
    ("meetpossum", ""),
    ("mintbaboon", ""),
    ("mintyeti", ""),
    ("proeagle", ""),
    ("propercoral", ""),
    ("pumpedcat", ""),
    ("strongman", ""),
    ("truecoyote", ""),
]


CASES_TO_IGNORE = (
    PROBLEMATIC_CASES
    + OTHER_VALIDATION_CASES
    + BALSAMIC_VALIDATION_CASES
    + FLUFFY_VALIDATION_CASES
    + MIP_VALIDATION_CASES
    + RNAFUSION_VALIDATION_CASES
    + TAXPROFILER_VALIDATION_CASES
)

def mark_validation_case(case_id: str, case_description: str, store: Store):
    """Update validation case in database."""

    case = store.get_case_by_internal_id(case_id)
    
    if case:
        case.is_compressible = False

        if case.comment and case_description:
            case.comment += f" {case_description}"
        else:
            case.comment = case_description
    else:
        print(f"Case {case_id} not found.")


if __name__ == "__main__":
    initialize_database("status_db_uri_here")
    store = Store()

    for case in CASES_TO_IGNORE:
        case_id, case_description = case
        mark_validation_case(case_id, case_description, store)

    store.session.commit()

[diitaz93]

Amazing 🔥

[islean]

Can you remove the validation ids in this PR? Would we not want "alembic migration -> set not compressible -> remove validation ids"?

[islean]

Looks nice 🤩

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[seallard]

Tested in stage and verified that the cases were updated as expected with

SELECT * FROM family WHERE internal_id IN (
  'causalmite', 'deepcub', 'expertalien', 'fluenteagle', 'grandkoi', 'lovingmayfly', 'loyalegret',
  'modernbee', 'proudcollie', 'richalien', 'suremako', 'wisestork',
  'setamoeba', 'sweetelf', 'poeticghoul', 'equalbug', 'stableraven', 'uphippo', 'cleanfowl',
  'proudsquid', 'modestjaguar', 'dearmarmot', 'holykid', 'civilsole', 'fleetjay', 'grandmarmot',
  'unitedbeagle', 'rightthrush', 'properpigeon', 'eagerox', 'casualweasel', 'acetuna', 'suitedsnake',
  'savinghorse', 'rightpup', 'sureroughy', 'notedshark', 'wholewhale', 'largeturtle', 'lightkodiak',
  'wholecivet', 'upwardstork', 'suitedgrub', 'simplesalmon', 'brightcaiman', 'casualgannet', 'civilkoala',
  'cleanshrimp', 'drivenmolly', 'easybeetle', 'epicasp', 'expertmole', 'finequagga', 'firstfawn',
  'gladthrush', 'helpedfilly', 'hotskink', 'inferret', 'intentcorgi', 'intentmayfly', 'justhusky',
  'kindcaiman', 'lightprawn', 'livingox', 'newaphid', 'nextjackal', 'modernmule', 'moralcattle',
  'onemite', 'opencow', 'proudcougar', 'rightmacaw', 'safeguinea', 'sharpparrot', 'sharppigeon',
  'sharpwhale', 'stillant', 'smoothboa', 'strongbison', 'tenderoriole', 'topsrhino', 'usablemarten',
  'vitalmouse', 'ableheron', 'acecoyote', 'ampleray', 'bossmink', 'cuddlyhen', 'daringowl', 'expertboar',
  'finerracer', 'growndoe', 'guidedfeline', 'handyturtle', 'hardygrouse', 'holyrodent', 'honestswine',
  'inlab', 'justburro', 'movedmule', 'nearbyjoey', 'oncrab', 'politeglider', 'rareosprey', 'rightmoray',
  'stablemoray', 'starnewt', 'tendergoose', 'truemole', 'truepigeon', 'valuedfowl', 'vocallocust',
  'wantedsawfly', 'richurchin', 'bigdrum', 'busycolt', 'daringpony', 'frankhusky', 'gamedeer',
  'hotviper', 'keencalf', 'keenviper', 'luckyhog', 'maturejay', 'meetpossum', 'mintbaboon',
  'mintyeti', 'proeagle', 'propercoral', 'pumpedcat', 'strongman', 'truecoyote'
);

Same in prod, looks fine. Deployed to Hasta prod.