More details required - how to change hyperparameters from CLI

[erotavlas]

Which page or section is this issue related to?

https://spacy.io/api/cli#train

It says

alias train-parser="python -m spacy train en /output /data /train /dev -n 1000"
token_vector_width=256 train-parser

However I tried this from the Windows command prompt and if you prepend the hyperparameters before calling python it complains that the command is not found.

Where is spacy reading these values from and how to set them on a Windows system.?

[svlandeg]

I think this comment in the docs refers to two things:

• You can set an alias for commands you run frequently - how you do that in your terminal depends on your system
• Throughout the 2.x code, spaCy allows setting hyperparameters from environment variables, as in the code here:

token_vector_width = util.env_opt('token_vector_width', cfg.get('token_vector_width', 96))

Which basically means that you can either set the argument values when calling a specific script, or set them as environment variables. In this case, the latter gets precedence as you can see from the code.

We've noticed that this practice has made it difficult to understand which parameters are set by which process and to which value, and as I just explained in your other issue, spaCy v.3 will deal with such hyperparameters quite differently by setting them specifically in a config file.

For now though in spaCy v.2, I think the most straightforward path is to just define the parameters you need as arguments for the script you're invoking. For instance, token_vector_width is set with the --width argument in train.cli, which is set to 96 by default.

cfg["token_vector_width"] = width

[erotavlas]

@svlandeg but what about parameters like dropout? I don't see this as an option as a cli argument. (although it seems to be an option for pretrain . Seems some are present others are not.

[svlandeg]

I'm afraid that for now, your best option is to look in the code itself:

dropout_rates = util.decaying(
        util.env_opt("dropout_from", 0.2),
        util.env_opt("dropout_to", 0.2),
        util.env_opt("dropout_decay", 0.0),
    )

[erotavlas]

@svlandeg to be clear I just wanted to verify if you are saying to find the matching CLI argument for the corresponding hyperparameter.

If that is true, dropout doesn't have a matching CLI argument I can use to set such value

It exists for pretrain but doesn't appear to be present for train so what is the alternative method to set that value? I'm just not understanding where I'm supposed to set these. (i.e. the ones for which there is no matching CLI argument possible as input to the CLI command)

def train(
    lang,
    output_path,
    train_path,
    dev_path,
    raw_text=None,
    base_model=None,
    pipeline="tagger,parser,ner",
    replace_components=False,
    vectors=None,
    width=96,
    conv_depth=4,
    cnn_window=1,
    cnn_pieces=3,
    use_chars=False,
    bilstm_depth=0,
    embed_rows=2000,
    n_iter=30,
    n_early_stopping=None,
    n_examples=0,
    use_gpu=-1,
    version="0.0.0",
    meta_path=None,
    init_tok2vec=None,
    parser_multitasks="",
    entity_multitasks="",
    noise_level=0.0,
    orth_variant_level=0.0,
    eval_beam_widths="",
    gold_preproc=False,
    learn_tokens=False,
    textcat_multilabel=False,
    textcat_arch="bow",
    textcat_positive_label=None,
    tag_map_path=None,
    omit_extra_lookups=False,
    verbose=False,
    debug=False,
):

[svlandeg]

Sorry that the current situation is confusing. Sometimes the parameters can be set through the CLI arguments, sometimes through environment variables, sometimes both options are possible and you'd actually have to check the code which takes precedence.

In the case of dropout for the train script, you can set it only with environment variables - cf the code I cited. These environment variables are read in util.py:

os.environ["SPACY_" + name.upper()]

I personally wouldn't really recommend setting these env. variables, unless you keep it well-documented on your end. You can find a list of them here.

You can ofcourse also tweak the local version of the train.py script, or implement your own training loop where you'd have much more freedom to set these parameters as you like.