Question about vocabulary size

[WENHUAN22]

Hi,

I tried to reimplement Delphi-2M on the UKBB dataset and to get patients' embeddings accordingly. While I am running the example_ukb_to_bin.py to prepare the train.bin and val.bin, I found that the icd10_codes_mod.tsv you provided contains 1129 records, after processing according to the script, the icdcodes contains 1148 records, and icdict contains 1171 records, neither of them equals 1269 (1270). Should the length of icdict be seen as the vocabulary size and equal to the length of label.csv?

How should we set the length of vocabulary size? And whether the file 'icd10_codes_mod.tsv' is the one you used for training the Delphi-2M?

Best regards,
Wenhuan

[yozhikoff]

Hi @WENHUAN22,

Thanks for your interest! Just to clarify - are you working in RAP or with a local copy of UKB?

We are currently working on a better version of the data processing script.
@tf2 Do you have a date in mind when it will be ready?

[WENHUAN22]

Hi @yozhikoff,

Thank you for the response. We are working with a local copy of UKB, the first occurrence data. Does it also work to train a Delph-2M model using a smaller set of ICD-10 codes, as what we can get in the 'icd10_codes_mod.tsv' file ?

[yozhikoff]

Sorry @WENHUAN22 I completely forgot to reply - is there still anything I can help you with?

[WENHUAN22]

Hi @yozhikoff, thank you for following up. Yes, we are currently trying to implement the model on the local copy of the UKBB dataset and obtain patient embeddings for further analysis. I want to make sure that we are not doing anything incorrectly while replicating the model. The mismatch in the number of ICD-10 codes raised my concern. Since we cannot obtain 1,269 ICD-10 disease tokens for training the model, I was wondering whether this could cause any problems.

[yozhikoff]

I will release the end-to-end training script for UKB RAP next week, covering both data preprocessing and model training.
Hopefully, this would help - the work in progress is already available in the ukb-rap branch. https://github.com/gerstung-lab/Delphi/blob/ukb-rap/data/ukb_real_data/example_ukb_rap_convert_new.ipynb

Regarding your initial question

the icdcodes contains 1148 records, and icdict contains 1171 records, neither of them equals 1269 (1270)

This is because they do not include cancer data, which is processed separately. In the notebook I linked above there is a dedicated section for them.

[WENHUAN22]

HI @yozhikoff, thank you for the quick response. Since we are currently working on a copy of the UKBB dataset stored on our HPC, the script for UKB RAP gives us a better understanding of the model replication process but does not help us directly. Our dataset uses field IDs (starting with f.), so the script you provided at data/simulated_data/example_ukb_to_bin.py has already been applied successfully to our data.

The icd10_codes_mod.tsv file provided in the same folder contains 1128 mappings (from field IDs to ICD codes). Could this be the reason for my question “the icdcodes contains 1148 records, and icdict contains 1171 records; neither of them equals 1269 (1270)”?

[yozhikoff]

Hi @WENHUAN22

Let me comment on where the number 1270 comes from.

Range	Count	Description
0–1	2	Technical tokens (Padding, No event)
2–3	2	Sex
4–12	9	Lifestyle (BMI, smoking, alcohol - 3 levels each)
13–1,269	1,257	ICD-10 codes (plus Death)

The ICD-10 tokens come from two sources:

• First-occurrence fields (icd10_codes_mod.tsv) - 1128 codes
• Cancer registry fields (f.40005/f.40006) - these add further ICD-10 codes that aren't in the first-occurrence fields

So to replicate with a local UKB copy, you need to process both the first-occurrence data (fields f.130000–f.132604) and the cancer registry (field f.40006 with dates from f.40005).

The cancer registry uses a different structure from the first-occurrence fields: instead of one dedicated
column per ICD-10 code, it stores cancer diagnoses as type/date pairs across instance indices - f.40006.{j}.0 (type) and f.40005.{j}.0 (date), with j from 0 to 16. j means first cancer occurrence, second, etc. This is why just a few columns give a lot of additional tokens.

To be precise, I just verified the numbers using the new RAP script.
icd10_codes_mod.tsv + Death gives 1130 tokens.
Lifestyle + sex gives 11.
The cancer registry data gives another 129.
Technical tokens give 2.

1130 + 11 + 129 + 2 = 1272... which is more than 1270 in the paper, because the paper uses older data.

[WENHUAN22]

Hi @yozhikoff, I appreciate your detailed explanation.

The ICD-10 tokens come from two sources:

• First-occurrence fields (icd10_codes_mod.tsv) - 1128 codes
• Cancer registry fields (f.40005/f.40006) - these add further ICD-10 codes that aren't in the first-occurrence fields

I have 1129 mappings from field IDs to ICD-10 (from your icd10_codes_mod.tsv). For constructing the cancer registry field IDs mappings, I used the codes from example_ukb_to_bin.ipynb:

for j in range(17):
icdict['f.40005.'+str(j)+'.0'] = "cancer_date_"+str(j)
icdict['f.40006.'+str(j)+'.0'] = "cancer_type_"+str(j)

Which resulted in 34 new mappings for cancer.

The cancer registry uses a different structure from the first-occurrence fields: instead of one dedicated column per ICD-10 code, it stores cancer diagnoses as type/date pairs across instance indices - f.40006.{j}.0 (type) and f.40005.{j}.0 (date), with j from 0 to 16. j means first cancer occurrence, second, etc. This is why just a few columns give a lot of additional tokens.

To be precise, I just verified the numbers using the new RAP script. icd10_codes_mod.tsv + Death gives 1130 tokens. Lifestyle + sex gives 11. The cancer registry data gives another 129. Technical tokens give 2.

Based on this, I think the mismatch is caused by how the cancer registry data are handled. Could you please explain a bit more about how the 129 cancer-related mappings are constructed from the registry data?

[WENHUAN22]

Hi @yozhikoff, after checking all.bin, I realized that I had some misunderstandings about the script, it contains more tokens than I initially thought. However, the minimum token value in all.bin is 1, and the maximum token value is 1268. I was wondering whether there is still an issue with my all.bin, since the maximum value is not 1270.

[yozhikoff]

@WENHUAN22
This is to be expected. The .bin file contains 1268 unique tokens (from 1 to 1268). The two remaining are technical - padding and no event - and will be added inside the get_batch() function.

Delphi/utils.py

Line 26 in f4ca07e

def get_batch(ix, data, p2i, select='center', index='patient', padding='regular',

[WENHUAN22]

Cool, thank you so much!

[WENHUAN22]

Hi @yozhikoff, I have another question, I found that there are 1256 (range from 1 to 1268) unique tokens in all.bin, specifically missing [7, 8, 9, 10, 11, 935, 781, 1038, 340, 1141, 1046, 1047], where the missing of lifestyle tokens makes me confused. After matching with label.csv, the lifestyle tokens that are missed in my all.bin are Smoking_mid, Smoking_high, Alcohol_low, Alcohol_mid, Alcohol_high.
Then I examined my UKBB phenotype dataset, the fields ID of smoking are recorded using the text {'Only occasionally', nan, 'No', 'Yes, on most or all days', 'Prefer not to answer'}, different from the text in labels.csv, same situation for the alcohol column. I wonder if we need additional processing for these two columns.

[yozhikoff]

@WENHUAN22
Yes, lifestyle tokens - all of them, BMI, alcohol and smoking- do need additional preprocessing.
We split them into bins (low, mid and high) based on some thresholds. Please look at the "Demographic and lifestyle data" section here:
https://github.com/gerstung-lab/Delphi/blob/ukb-rap/data/ukb_real_data/example_ukb_rap_convert_new.ipynb

For the other missing tokens, I can't be sure, but it may be due to different data versions. As a sanity check, you can compare your token counts with counts from here:
https://github.com/gerstung-lab/Delphi/blob/ukb-rap/delphi_labels_chapters_colours_icd.csv