DocBin.to_bytes fails with a "ValueError: bytes object is too large" Spacy v 3.7.2

[rajesh-smartwebtech]

How to reproduce the behaviour

I am trying to train a model from scratch (NER) on a corpus which has around 8 million sentences, after added data in DocBin() then unable to save it getting error

Your Environment

• Operating System: WIN 10 (64bit)
• Python Version Used: 3.9.0
• spaCy Version Used: 3.7.2
• Environment Information:

I have tried to make chunk of DocBin then murge and save single file but same issue

import os
import spacy
from spacy.tokens import DocBin
from tqdm import tqdm
from spacy.util import filter_spans

merged_doc_bin = DocBin()

fiels = [
    "G:\\success-demo\\product_ner\\test\\train3.spacy",# 3000000 tokens here
    "G:\\success-demo\\product_ner\\test\\train1.spacy", # 3000000 tokens here
    "G:\\success-demo\\product_ner\\test\\train2.spacy", # 2000000 tokens here
]

for filename in fiels:
    doc_bin = DocBin().from_disk(filename)
    merged_doc_bin.merge(doc_bin)

merged_doc_bin.to_disk("G:\\success-demo\\product_ner\\test\\final\\murge.spacy")

[adrianeboyd]

Copied from https://stackoverflow.com/a/77690259:

This is a limitation due to msgpack, where a single file can't be larger than 2GB.

You don't need to merge these files, though. You can provide a directory rather than a single .spacy file for spacy train:

spacy train config.cfg --paths.train train/ --paths.dev dev/

It will recursively load all .spacy files in the directory.