Manifest content hash computation times out

[dsotirho-ucsc]

Computing content hash for manifest took over 30 seconds. A reindex was not occurring at this time.

CloudWatch logs (anvilprod):

[
    {
        "@timestamp": "2024-04-02 18:22:46.909",
        "@message": "START RequestId: 25cf6787-dc1c-415f-a5c2-bd788d467dfb Version: $LATEST\n"
    },
…
    {
        "@timestamp": "2024-04-02 18:22:46.925",
        "@message": "[DEBUG]\t2024-04-02T18:22:46.925Z\t25cf6787-dc1c-415f-a5c2-bd788d467dfb\t
                     azul.service.manifest_service\tComputing content hash for manifest using 
                     filters Filters(explicit={
                         'donors.organism_type': {
                             'is': ['Homo sapiens', None]
                         }, 
                         'files.file_format': {
                             'is': ['.vcf.gz', '.cram', '.crai', '.tbi', '.md5', '.idat', '.csi',
                                    '.gtc', '.tar', '.bam', '.svs', '.bai', '.txt', '.tab.gz', 
                                    '.bigWig', '.junc.gz', '.fastq.gz', '.idat.gz', '.csv', 
                                    '.bed.gz', '.tsv.gz', '.gvcf.gz', '.interval_list', '.txt.gz',
                                    '.bw', '.bedpe', '.alignment_summary_metrics', 
                                    '.arrays_control_code_summary_metrics', '.loupe', 
                                    '.bait_bias_detail_metrics', '.bait_bias_summary_metrics', 
                                    '.crosscheck', '.duplicate_metrics', '.insert_size_metrics', 
                                    '.pre_adapter_detail_metrics', '.pre_adapter_summary_metrics', 
                                    '.quality_distribution_metrics', '.fingerprinting_summary_metrics', 
                                    '.h5', '.detail_metrics', '.summary_metrics', '.tsv', '.mtx', 
                                    '.md5sum', '.bcf', '.vcf', '.vcf.bgz', '.raw_wgs_metrics', 
                                    '.wgs_metrics', '.cloupe', '.html', '.ped', '.log', '..gz', '.bed', 
                                    '.wdl', '.bb', '.egt', '.xlsx', '.tar.gz', None, '.gct.gz', 
                                    '.variant_calling_summary_metrics', '.R', '.fai', '.fasta', 
                                    '.h5ad', '.dict', '.jpg', '.md', '.pdf', '.readme', '.chain', 
                                    '.docx', '.fingerprintcheck', '.gct', '.sizes', 
                                    '.variant_calling_detail_metrics', '.xls', '.zip', '.bim', '.fam', 
                                    '.gff3.gz', '.gtf', '.gtf.gz', '.ipynb', '.parquet', '.png']
                         }
                     }, 
                     source_ids={
                         '0af50ad2-1e53-4cc1-ba71-280b04a9f488',
                         'df0b99c1-1b1d-482d-930c-dea8d6101f78',
                         '3047ec0b-695e-4c1f-8c20-bf060b5f72aa',
                         '1167858d-f9d9-4dbc-bbcb-473ac21eb823',
                         '2f8a866a-b38a-447c-b2d4-a22d998e659a',
                         '3ac713b5-3645-4381-ac66-ecbc281a2ab8',
                         …
                         "
    },
    {
        "@timestamp": "2024-04-02 18:22:46.929",
        "@message": "[INFO]\t2024-04-02T18:22:46.929Z\t25cf6787-dc1c-415f-a5c2-bd788d467dfb\t
                     elasticsearch\tMaking POST request to
                     https://vpc-azul-index-anvilprod-ggipah4skn2ftt47u4xgvydzqm.us-east-1.es.amazonaws.com:443/azul_v2_anvilprod_anvil4_files_aggregate/_search"
    },
    {
        "@timestamp": "2024-04-02 18:22:46.929",
        "@message": "[INFO]\t2024-04-02T18:22:46.929Z\t25cf6787-dc1c-415f-a5c2-bd788d467dfb\t
                     elasticsearch\t… with request body b'{\"query\":{\"bool\":{\"must\":[{
                     \"constant_score\":{\"filter\":{\"bool\":{\"should\":[{\"terms\":{
                     \"contents.donors.organism_type.keyword\":[\"...'"
    },
    {
        "@timestamp": "2024-04-02 18:23:17.948",
        "@message": "2024-04-02T18:23:17.948Z 25cf6787-dc1c-415f-a5c2-bd788d467dfb 
                     Task timed out after 31.04 seconds\n\n"
    },
    {
        "@timestamp": "2024-04-02 18:23:17.948",
        "@message": "END RequestId: 25cf6787-dc1c-415f-a5c2-bd788d467dfb\n"
    },
    {
        "@timestamp": "2024-04-02 18:23:17.948",
        "@message": "REPORT RequestId: 25cf6787-dc1c-415f-a5c2-bd788d467dfb\t
                     Duration: 31038.88 ms\tBilled Duration: 31000 ms\tMemory Size: 2048 MB\tMax Memory Used: 239 MB\t\n"
    }
]

[dsotirho-ucsc]

Assignee to provide description.

[hannes-ucsc]

Can you please put the description into a form that doesn't require horizontal scroll to see the meaningful parts?

[achave11-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

I plotted the hash computation durations for the last four weeks in prod:

… and anvilprod

For HCA, it looks like the computation is usually quick and that the outliers are only half of the Lambda timeout. For AnVIL, fewer manifests are being requested so the graph is fairly sparse. One of the outliers got close to the timeout, and occurred two days after the reproduction.

I am not too worried about this at the moment but will post a proposed patch shortly.

[hannes-ucsc]

Index: src/azul/service/manifest_service.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/service/manifest_service.py b/src/azul/service/manifest_service.py
--- a/src/azul/service/manifest_service.py	(revision eff8bddbaceeb49de5350c38a62f6c32a1f71fcb)
+++ b/src/azul/service/manifest_service.py	(date 1714519929425)
@@ -999,8 +999,16 @@
         """
         git_commit = config.lambda_git_status['commit']
         filter_string = repr(sort_frozen(freeze(self.filters.explicit)))
-        content_hash = str(self.manifest_content_hash)
         catalog = self.catalog
+        # PATCH: Implement this property and the backing environment variable
+        # defaulting to 0. Patch the variable to 1 for unittests. Enforce it by
+        # rejecting bundle notifications in the indexer API if the variable is 0
+        if config.enable_bundle_notifications:
+            content_hash = str(self.manifest_bundles_hash)
+        else:
+            # If we incremental index changes are disabled, we don't need to
+            # worry about individual bundles, only sources.
+            content_hash = str(self.manifest_sources_hash)
         format = self.format()
         manifest_hash_input = [
             git_commit,
@@ -1162,7 +1170,33 @@
                                           **args))
 
     @cached_property
-    def manifest_content_hash(self) -> int:
+    def manifest_sources_hash(self) -> int:
+        """
+        Return a hash of the identifiers of the sources from which
+        projects/datasets containing files matching the current filters were
+        indexed. It's worth noting that a filter may match a project/dataset but
+        none of the project's files. For example, if a project contains only
+        files derived from either mouse brains or lion hearts, the project will
+        match the filter `species=lion and organ=brain`, but none of its files
+        will. If such a project/dataset is added/removed to/from the index, the
+        manifest hash returned for a given filter will be different even though
+        the contents of the manifest hasn't changed, as no matching files were
+        added or removed.
+
+        So while this hash is less sensitive than the one returned by
+        :py:meth:`manifest_bundles_hash`, it can be computed much more quickly.
+        """
+        # PATCH: This should go against the projects/datasets index
+
+    @cached_property
+    def manifest_bundles_hash(self) -> int:
+        """
+        Return a hash of the fully-qualified identifiers of all bundles
+        containing files that match the current filter. The return value
+        approximates a hash of the content of the manifest because a change of
+        the file data requires a change to the file metadata which requires a
+        new bundle or bundle version.
+        """
         log.debug('Computing content hash for manifest using filters %r ...', self.filters)
         start_time = time.time()
         request = self._create_request()