Spacy high memory consumption issue

[nikhilcms]

Hello,
I am running spacy model with english medium weights inside kubernetes pod.

As I am observing after loading spacy model, it's taking around 500mb space and while after every prediction it keeps increasing.

I am wondering even after deleting spacy object then also it's not releasing memory.

I have allotted around 1GB memory resources to my pod, but after few hours it consume complete memory and stuck pod.

Could you please provide solutions how to release memory and ensure memory should not increase with number of predictions

[adrianeboyd]

Here's a thread that covers the memory usage related to the vocab: #10015

If you're only using a md model which doesn't use pytorch, you might be able to reduce the memory usage on load by removing torch from your venv. The torch module is automatically loaded if it's available by thinc, and if RAM is very limited skipping torch could help?

[nikhilcms]

Hi @adrianeboyd , I have not installed torch module to use spacy.

exclude = [ "parser","textcat","tagger","tokenizer",]
nlp = spacy.load("en_core_web_md", exclude=exclude)

I am using above code to load spacy , as I observed from logs english model taking around 550-600MB pod memory.
and it works fine for smaller and medium query (slightly memory gets increasing as number of requests to spacy model increases, but slight increase fine for me).
but when we are passing longer query (text contains more than 2500 token (split by space)) to spacy model, it takes around 300MB extra memory and its not released after inference.
How to address this issue ?

[adrianeboyd]

Which components / which annotations do you need from en_core_web_md?

[nikhilcms]

I just wanted to user 'ner' from en_core_web_md.

[adrianeboyd]

The ner component in en_core_web_md doesn't depend on any other components, so you can disable everything nlp.pipe_names except for ner.

I think you're mainly noticing that python doesn't necessarily release memory immediately.

Longer docs will use more memory, especially for the parser or ner components. Since ner annotation doesn't typically depend on a ton of context, you can usually split up long docs for processing on paragraph or section boundaries without much difference in accuracy, although 2500 tokens isn't particularly long. If you need to rejoin smaller docs to a single doc, you can use Doc.from_docs.

[nikhilcms]

Thanks @adrianeboyd , yeah I have exclude other components and I found rather than passing long text, if we pass splited sentences memory consumption reduced, but still memory leakage is there... pod memory increases slowly. as number of request increases it may exceed highest limit.
there should be a way to stabilize memory irrespective of number of request.
please have a look below code if we can optimize further !

`
import spacy
from spacy.tokens import Doc
from nltk.tokenize.punkt import PunktSentenceTokenizer

detected_sentences = []
for start, end in PunktSentenceTokenizer().span_tokenize(text_):
text_tuple = text_[start:end]
detected_sentences.append(text_tuple)

nlp = spacy.load("en_core_web_md", exclude=['tok2vec', 'tagger', 'parser', 'senter', 'attribute_ruler', 'lemmatizer'])

docs = list(nlp.pipe(detected_sentences))
c_doc = Doc.from_docs(docs)
entity_list = [(ent,ent.start_char,ent.end_char) for ent in c_doc.ents]
`

[adrianeboyd]

Please read the thread linked in the first response above.

[reachusama]

I am using the below code for loading spacy and it is getting memory intensive as the text size increases. Is there a way to reduce memory usage or limit the length of text for specific ram usage ?

import pytextrank
import spacy
from spacy.tokens import Span


@spacy.registry.misc("combined_scrubber")
def combined_scrubber():
    def scrubber_func(span: Span) -> str:
        first_non_det_pron = next(
            (
                index
                for index, token in enumerate(span)
                if token.pos_ not in ["DET", "PRON"]
            ),
            len(span),
        )
        new_span = span[first_non_det_pron:]
        cleaned_span = " ".join(
            [
                token.text
                for token in new_span
                if not token.is_stop and not token.is_punct and not token.like_num
            ]
        )
        return cleaned_span

    return scrubber_func


nlp = spacy.load("en_core_web_sm")
nlp.add_pipe(
    "topicrank",
    config={
        "stopwords": {"word": ["NOUN"]},
        "scrubber": {"@misc": "combined_scrubber"},
    },
)