Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variation of scores of different runs of inference.py #260

Open
sri480673 opened this issue Oct 24, 2024 · 0 comments
Open

Variation of scores of different runs of inference.py #260

sri480673 opened this issue Oct 24, 2024 · 0 comments

Comments

@sri480673
Copy link

sri480673 commented Oct 24, 2024

Hi @gcorso,
I have tested for CPU enabled inference.py of recent version(1.1.3) of Diffdock, where the output scores are different by rerunning the inference.py And number of failed cases also different.

Below shows the two run outputs:
ls results_batch_size_10/user_predictions_small/5zcu/
rank10_confidence-3.96.sdf rank2_confidence-0.51.sdf rank5_confidence-1.50.sdf rank8_confidence-3.18.sdf
rank1_confidence-0.37.sdf rank3_confidence-0.53.sdf rank6_confidence-1.71.sdf rank9_confidence-3.21.sdf

ls results_batch_size_10_rerun/user_predictions_small/5zcu:
rank10_confidence-4.67.sdf rank2_confidence-0.34.sdf rank5_confidence-0.64.sdf rank8_confidence-1.94.sdf
rank1_confidence-0.12.sdf rank3_confidence-0.44.sdf rank6_confidence-1.92.sdf rank9_confidence-3.66.sdf
rank1.sdf rank4_confidence-0.49.sdf rank7_confidence-1.92.sdf

After I set the no_random=True in randomize_position() and sampling() called from inference.py Still getting variation in scores by rerunning the inference.py and different no. of failed cases also.

Below is output of one of the score of protein 5zcu,
rank10_confidence-4.34.sdf rank2_confidence-1.38.sdf rank5_confidence-2.71.sdf rank8_confidence-3.67.sdf
rank1_confidence-1.35.sdf rank3_confidence-2.27.sdf rank6_confidence-2.92.sdf rank9_confidence-3.80.sdf
rank1.sdf rank4_confidence-2.48.sdf rank7_confidence-3.65.sdf

ls results_1_norandom/user_predictions_small/5zcu:
rank10_confidence-4.07.sdf rank2_confidence-2.11.sdf rank5_confidence-2.78.sdf rank8_confidence-3.71.sdf
rank1_confidence-1.86.sdf rank3_confidence-2.30.sdf rank6_confidence-3.30.sdf rank9_confidence-3.81.sdf
rank1.sdf rank4_confidence-2.30.sdf rank7_confidence-3.60.sdf

Along with no_random=True and fix the seed for numpy,torch and random package, still getting different failed cases as well as scores are also different.
Please give a suggestion how to overcome the above issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant