[SDK] ValueError: <HUB_TOKEN> is not a valid HubStrategy, please select one of ['end', 'every_save', 'checkpoint', 'all_checkpoints']

[mahdikhashan]

What happened?

when trying to create a tune job to optimize hp of llm(s), with the below values for tune api, i received an unclear error - it could be better when it shows be what has been wrong.

hf_model = HuggingFaceModelParams(
    model_uri = "hf://meta-llama/Llama-3.2-1B",
    transformer_type = AutoModelForSequenceClassification,
)

# Train the model on 1000 movie reviews from imdb
# https://huggingface.co/datasets/stanfordnlp/imdb
hf_dataset = HuggingFaceDatasetParams(
    repo_id = "imdb",
    split = "train[:1000]",
)

hf_tuning_parameters = HuggingFaceTrainerParams(
    training_parameters = TrainingArguments(
        output_dir = "results",
        save_strategy = "no",
        hub_strategy="all_checkpoints",
        learning_rate = 1e-05, #katib.search.double(min=1e-05, max=5e-05),
        num_train_epochs=3,
    )
)

cl = KatibClient(namespace="kubeflow")

exp_name = "testllm"

cl.tune(
    name = exp_name,
    model_provider_parameters = hf_model,
    dataset_provider_parameters = hf_dataset,
    trainer_parameters = hf_tuning_parameters,
    objective_metric_name = "train_loss",
    objective_type = "minimize",
    algorithm_name = "random",
    max_trial_count = 10,
    parallel_trial_count = 2,
    resources_per_trial={
        "gpu": "2",
        "cpu": "4",
        "memory": "10G",
    },
)

traceback

ValueError                                Traceback (most recent call last)
ValueError: '<HUB_TOKEN>' is not a valid HubStrategy

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[19], line 3
      1 # Fine-tuning for Binary Classification
      2 exp_name = "testllm"
----> 3 cl.tune(
      4     name = exp_name,
      5     model_provider_parameters = hf_model,
      6     dataset_provider_parameters = hf_dataset,
      7     trainer_parameters = hf_tuning_parameters,
      8     objective_metric_name = "train_loss",
      9     objective_type = "minimize",
     10     algorithm_name = "random",
     11     max_trial_count = 10,
     12     parallel_trial_count = 2,
     13     resources_per_trial={
     14         "gpu": "2",
     15         "cpu": "4",
     16         "memory": "10G",
     17     },
     18 )
     20 cl.wait_for_experiment_condition(name=exp_name)
     22 # Get the best hyperparameters.

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/site-packages/kubeflow/katib/api/katib_client.py:602, in KatibClient.tune(self, name, model_provider_parameters, dataset_provider_parameters, trainer_parameters, storage_config, objective, base_image, parameters, namespace, env_per_trial, algorithm_name, algorithm_settings, objective_metric_name, additional_metric_names, objective_type, objective_goal, max_trial_count, parallel_trial_count, max_failed_trial_count, resources_per_trial, retain_trials, packages_to_install, pip_index_url, metrics_collector_config)
    600 experiment_params = []
    601 trial_params = []
--> 602 training_args = utils.get_trial_substitutions_from_trainer(
    603     trainer_parameters.training_parameters, experiment_params, trial_params
    604 )
    605 lora_config = utils.get_trial_substitutions_from_trainer(
    606     trainer_parameters.lora_config, experiment_params, trial_params
    607 )
    609 # Create the init and the primary container.

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/site-packages/kubeflow/katib/utils/utils.py:207, in get_trial_substitutions_from_trainer(parameters, experiment_params, trial_params)
    205                 value = copy.deepcopy(p_value)
    206             else:
--> 207                 value = type(old_attr)(p_value)
    208         setattr(parameters, p_name, value)
    210 if isinstance(parameters, TrainingArguments):

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/enum.py:384, in EnumMeta.__call__(cls, value, names, module, qualname, type, start)
    359 """
    360 Either returns an existing member, or creates a new enum class.
    361 
   (...)
    381 `type`, if set, will be mixed in as the first base class.
    382 """
    383 if names is None:  # simple value lookup
--> 384     return cls.__new__(cls, value)
    385 # otherwise, functional API: we're creating a new Enum type
    386 return cls._create_(
    387         value,
    388         names,
   (...)
    392         start=start,
    393         )

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/enum.py:709, in Enum.__new__(cls, value)
    704             exc = TypeError(
    705                     'error in %s._missing_: returned %r instead of None or a valid member'
    706                     % (cls.__name__, result)
    707                     )
    708         exc.__context__ = ve_exc
--> 709         raise exc
    710 finally:
    711     # ensure all variables that could hold an exception are destroyed
    712     exc = None

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/enum.py:692, in Enum.__new__(cls, value)
    690 try:
    691     exc = None
--> 692     result = cls._missing_(value)
    693 except Exception as e:
    694     exc = e

File ~/miniconda3/envs/llm-hp-optimization-katib-nb/lib/python3.9/site-packages/transformers/utils/generic.py:498, in ExplicitEnum._missing_(cls, value)
    496 @classmethod
    497 def _missing_(cls, value):
--> 498     raise ValueError(
    499         f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}"
    500     )

ValueError: <HUB_TOKEN> is not a valid HubStrategy, please select one of ['end', 'every_save', 'checkpoint', 'all_checkpoints']

What did you expect to happen?

a clear value error message to mention which field is wrong - and what is the expected value.

Environment

Kubernetes version:

$ kubectl version

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"

Katib Python SDK version:

$ pip show kubeflow-katib

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[mahdikhashan]

/area-sdk

[google-oss-prow]

@helenxie-bit: The label(s) /remove-label lifecycle/need-triage cannot be applied. These labels are supported: tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, lifecycle/needs-triage. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/remove-label lifecycle/need-triage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[helenxie-bit]

@mahdikhashan Hi, could you try installing Python 3.12 to see if this resolves the issue? I'm currently using Python 3.12 and haven't encountered this error. Since the line value = type(old_attr)(p_value) uses a Python built-in construct, I suspect the issue might be related to the Python version.

[helenxie-bit]

/remove-label lifecycle/needs-triage

[helenxie-bit]

/area sdk

[mahdikhashan]

@mahdikhashan Hi, could you try installing Python 3.12 to see if this resolves the issue? I'm currently using Python 3.12 and haven't encountered this error. Since the line value = type(old_attr)(p_value) uses a Python built-in construct, I suspect the issue might be related to the Python version.

I'll do so and share the results. Thanks for your reply. Stay tuned.

[mahdikhashan]

while I was checking the python version on my setup (it is 3.9 atm), I noticed two things:

in the api document, the requirements for the python version is asserted to be python >= 3.8 (https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md?plain=1#L7) and in the ci test it is using 3.11 (https://github.com/kubeflow/katib/blob/master/.github/workflows/test-python.yaml#L24)

please correct me when I'm wrong @helenxie-bit

do we want to increase the required python version (which may cause breaking changes)? or ?

update:

actually my opinion for ci is to use a matrix testing and support from version 3.8 above -

[helenxie-bit]

while I was checking the python version on my setup (it is 3.9 atm), I noticed two things:

in the api document, the requirements for the python version is asserted to be python >= 3.8 (https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md?plain=1#L7) and in the ci test it is using 3.11 (https://github.com/kubeflow/katib/blob/master/.github/workflows/test-python.yaml#L24)

please correct me when I'm wrong @helenxie-bit

do we want to increase the required python version (which may cause breaking changes)? or ?

update:

actually my opinion for ci is to use a matrix testing and support from version 3.8 above -

I think the python version for ci test is 3.10.

I agree that we should test with python versions that match the requirements in the documentation. But let’s first test the example in the LLM hyperparameter optimization API to ensure it works successfully. Based on the results, we can update the documentation or code as needed.

Also, there might be other issues to address, like the one I mentioned here.

[mahdikhashan]

while I was checking the python version on my setup (it is 3.9 atm), I noticed two things:
in the api document, the requirements for the python version is asserted to be python >= 3.8 (https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md?plain=1#L7) and in the ci test it is using 3.11 (https://github.com/kubeflow/katib/blob/master/.github/workflows/test-python.yaml#L24)
please correct me when I'm wrong @helenxie-bit
do we want to increase the required python version (which may cause breaking changes)? or ?
update:
actually my opinion for ci is to use a matrix testing and support from version 3.8 above -

I think the python version for ci test is 3.10.

I agree that we should test with python versions that match the requirements in the documentation. But let’s first test the example in the LLM hyperparameter optimization API to ensure it works successfully. Based on the results, we can update the documentation or code as needed.

Also, there might be other issues to address, like the one I mentioned here.

hm, i meant the unit tests ci, its fine, lets leave it for later.