1) Model performance, 2) Imbalanced behavior dataset

[histun]

Hi. Thank you for this amazing package. I'm trying to use cebra with neural activity (4500, n_neurons) + annotated social behavior pattern variables.

1. Model performance
   I have trained multipled models with 1) speed of the experimental animal, 2) social distance, 3) individual behavior (rearing, sniffing, etc) 4) social behavior (investigating mouse2, etc), 5) shuffled social (divided into 5 blocks and shuffled), 6) 50block shuffled
   I cannot understand why the shuffled model's loss is still so good. If I shuffle a lot more, then the loss becomes bad.
   

Below is the AUC from the decoder built with 4) social behaivor and 5) shuffled social behavior. It looks like the decoding performance of the shuffled dataset is a lot worse, but I can't think of why the loss is similar in the embedding.

2. Imbalanced dataset
   I have divided the social behavior datset (4500,) with 1-10 int values into another variable (4500,10), thinking that I could train ten different models with each social annotation, and compare each behavior (below). However, this seems to reflect the imbalanced datset where I have the most mouse1_strong and no_int. I might be thinking not correctly, but is this a viable approach? If so, how do I avoid this issue? subsampling mouse1_strong and no_int to match the lowest occuring behavior will lose a lot of data.
   

[stes]

Hi @histun , thanks for asking --- could you provide more details (maybe a code snippet) on how you do the shuffling? Can you also unpack the sentence "If I shuffle a lot more, then the loss becomes bad."?

[histun]

`
#Shuffling in blocks

n_blocks = 50

Determine the size of each block (note: the last block might be smaller)

block_size = len(MMs_major_data1) // n_blocks

Create a list to hold the blocks

blocks = [MMs_major_data1[i*block_size : (i+1)*block_size] for i in range(n_blocks - 1)]
blocks.append(MMs_major_data1[(n_blocks-1)*block_size : ])

Shuffle the order of the blocks

np.random.shuffle(blocks)

Concatenate the blocks back into a single array

shuffled1_MMs_major_data1 = np.concatenate(blocks)`

Shuffling more means increaing the value of 'n_blocks,' which then leads to higher loss value. I would have expected even n_blocks = 5 would lead to significant increase in the loss value like random shuffling, but it didn't change much.