Updating Trained Textcat Model With New Data

[creamofclubs]

I am currently working on updating a Text Classification model and am seeking your assistance.

Spacy 3.7

The initial model construction process took approximately 15 hours and yielded significant success. However, I have encountered a challenge in the subsequent update. Despite my efforts to incorporate additional data for improved accuracy, the updated model appears to only recognize the new data during testing.

I find myself genuinely perplexed by this issue and have exhaustively searched for solutions without success. Your expertise and guidance on this matter would be immensely valuable.

Thank you in advance for your time and assistance.

MODELFLD = 'model/loc/'

def train_spacy_textcat_json(data,iterations ,model_output_path):
    nlp = spacy.load(f'{MODELFLD}model-best')
    TRAIN_DATA = data
    
    if 'textcat' not in nlp.pipe_names:
        textcat = nlp.create_pipe('textcat')
        nlp.add_pipe(textcat, last=True)
    else:
        textcat = nlp.get_pipe('textcat')
    
    # Add labels to the text classification model
    for entry in TRAIN_DATA:
        label = entry['category']
        textcat.add_label(label)

    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']
    
    with nlp.disable_pipes(*other_pipes):  # Only train textcat
        optimizer = nlp.begin_training()
        
        for itn in range(iterations):
            print("Starting iteration " + str(itn))
            random.shuffle(TRAIN_DATA)
            losses = {}
            
            for entry in TRAIN_DATA:
                text = entry['sentence']
                label = entry['category']
                
                golds = {label: 1.00}  # Dictionary with the correct label and probability
                
                example = Example.from_dict(nlp.make_doc(text), {'cats': golds})
                
                nlp.update(
                    [example],  # Batch of examples
                    drop=0.2,
                    losses=losses,
                )
            
            print(losses)
    nlp.to_disk(model_output_path)
    return nlp

train_spacy_textcat_json(training_data,20,"./Models/j7cat")

[svlandeg]

The initial model construction process took approximately 15 hours and yielded significant success.

Nice work!

However, I have encountered a challenge in the subsequent update. Despite my efforts to incorporate additional data for improved accuracy, the updated model appears to only recognize the new data during testing.

If I understand you correctly, you've been updating your model with new data, which made it forget its "old" capabilities? This sounds like the typical issue of catastrophic forgetting to me. Basically what happens is that the model indeed updates its weights according to the (new) information you're giving it - and it won't automagically remember the "old" training data. Instead, what you should do in these cases is to make sure that you keep feeding the models "old" examples among the new ones, so that it will still get penalized if it gets those cases wrong, and ultimately will succeed in performing well on both datasets.