Extremely high consistency score across datasets for discrete label

[Jinwoo-Yi]

Dear CEBRA team,

I am currently preparing my thesis using your excellent package, for which I am deeply grateful.

In my work, embeddings were learned based on contrasting discrete labels (including four levels). To calculate the consistency score across datasets, I set n_bins = 4, following the recent discussion here. However, I encountered an issue where all pairs' consistency scores were estimated as 1, an extremely high R-squared value.

Additionally, I observed the same result when calculating this metric using (1) pseudo-CEBRA embeddings with permuted discrete labels and (2) UMAP embeddings. I have attached the script used to compute the consistency score for your reference.

Could you assist me in identifying any potential issues here?

Thank you very much for your continuous support and active interactions with users.

Sincerely,
Jinwoo

[stes]

Hi @Jinwoo-Yi , this is due to the low number of bins, n_bins = 4. How many different discrete labels do you have? The metric becomes "harder" the higher the number of bins.

[Jinwoo-Yi]

@stes thanks for your reply. In my dataset, the auxiliary variable has four discrete labels, so I defined n_bins = 4 considering recent discussion (e.g., #125). Is it okay to set higher values as n_bins in this case? then which value will be appropriate? (P.S. In the attached screenshot, n_bins = 12 but this is a just typo. I confirmed that the results I described originated from n_bins = 4).

[Barbod1380]

I am encountering a similar issue when trying to compute the consistency score across 12 sessions. Each session involves the presentation of two image frames, either labeled as 3 or 8. As a result, my labels are structured as a 2D array with 12 rows (one for each session) and approximately 1000 columns, where each element is either 3 or 8.

The problem arises when I set num_discretization_bin to 12. I receive the following error:

ValueError: Digitalized labels do not have elements close enough to bin index 4. The bin index should be in the range of the labels' values.

When I set the num_discretization_bin to 2, I receive a warning:

UndefinedMetricWarning: R^2 score is not well-defined with less than two samples.

In this case, the consistency score does not generate any meaningful result. For the range between 3 and 6, all consistency scores are 99%.

Could you please help clarify what might be causing these issues and how I could adjust the parameters to compute the consistency score correctly?

[stes]

Hi @Barbod1380 , thanks for the follow up Q. The issue is the same as above. If you only have 12 bins, you collapse the whole embedding into the average for these 12 bins. This means that it is considerably easier to get a high consistency score vs. increasing the number of bins to a larger value, so what you are seeing is expected behavior.

Could you post an image of the embeddings you are getting, and also provide a histogram of the labels you are feeding into the consistency score?

[Barbod1380]

"I reduced the number of training sessions from 12 to 6. Now, for each of these 6 training sessions, I have a discrete_label that can either be '3' or '8'. The distribution of labels is consistent across all sessions, meaning that each session contains an equal number of '3' and '8' labels.

This image shows my embeddings, where the blue-colored embedding corresponds to the label '3', and the red-colored one represents the label '8'. Given this, it seems more reasonable to have a 99% consistency score across sessions, though I'm not entirely certain about it.

[stes]

Hi @Barbod1380 , thanks for sharing these. So as you see here, the consistency will be very high because all the model needs to do is to properly align these two blobs of data.

Without knowing too much about your data, I would suggest to consider if there are any other variables you could align to which are continuous. If you include such a continuous variable, you will see some kind of continuous structure in the embedding (vs. only a blob of data) which can then be compared using the consistency metric. This could be something as simple as the trial time. If you then train an embedding using either the joint continous-discrete mode, or the continous mode, the metric would give you a more meaningful interpretation of the data.

Would that be applicable in your setup?