Possible missing data of benchmarking supervised performance

[Eikor]

Hi, thank you for such fundamental work for the computational bio community!

I am interested in evaluating the model's performance in a supervised setting.
I have download the DMS_supervised_substitutions_scores.csv and run the script provide in scripts/scoring_DMS_supervised/performance_substitutions.sh, I modified the --input_scoring_file as DMS_supervised_substitutions_scores.csv and --DMS_reference_file_path as DMS_substitutions.csv. However, there is an error occurred in

ProteinGym/proteingym/performance_DMS_supervised_benchmarks.py

Line 53 in 495cc30

old_ids = ref_df["Old_DMS_ID"].unique()

, I wonder am I passing the wrong reference file or this script needs to be updated with current reference file?

Thanks!

[pascalnotin]

Hi @Eikor - thank you for your question!

The error you are experiencing is due to a change in naming conventions for certain of the files (also discussed here), and we were indeed using a different reference file (similar to this one but with a mapping to the old IDs as well) when running this particular script internally. We will fix these issues in a new release as soon as possible.
In the meantime, you may use the performance files here which are the ones you would obtain by running this script.

Kind regards,
Pascal