[WIP] Training: Initial Documentation for Kubeflow Trainer V2

[andreyvelich]

Fixes: kubeflow/training-operator#2214

This is initial version for Kubeflow Training V2 docs.
Please let me know what do you think.

TODOs:

• Add working Getting Started example.
• Fix the installation scripts.
• Add new logo of Kubeflow Training.
• Rename Kubeflow Training Operator -> Kubeflow Training everywhere.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @hbelmiro @varodrig @jbottum @varshaprasad96 @akshaychitneni @helenxie-bit @Electronic-Waste @saileshd1402 @seanlaii @deepanker13 @astefanutti @shravan-achar @kannon92 @droctothorpe @sandipanpanda @vsoch @franciscojavierarceo @Syulin7 @StefanoFioravanzo @kuizhiqing

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: astefanutti, kannon92, shravan-achar, vsoch, kubeflow/wg-training-leads, kubeflow/release-team, akshaychitneni, seanlaii, varshaprasad96, saileshd1402.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Fixes: kubeflow/training-operator#2214

This is initial version for Kubeflow Training V2 docs.
Please let me know what do you think.

TODOs:

• Add working Getting Started example.
• Fix the installation scripts.
• Add new logo of Kubeflow Training.
• Rename Kubeflow Training Operator -> Kubeflow Training everywhere.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @hbelmiro @varodrig @jbottum @varshaprasad96 @akshaychitneni @helenxie-bit @Electronic-Waste @saileshd1402 @seanlaii @deepanker13 @astefanutti @shravan-achar @kannon92 @droctothorpe @sandipanpanda @vsoch @franciscojavierarceo @Syulin7 @StefanoFioravanzo @kuizhiqing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

Let me know if that message looks good to you @kubeflow/wg-training-leads @rimolive @varodrig @hbelmiro @StefanoFioravanzo.
If yes, I will add it to all Kubeflow Training V1 docs, similar to KFP: https://www.kubeflow.org/docs/components/pipelines/legacy-v1/overview/quickstart/

[Electronic-Waste]

@andreyvelich Great contributions! I left some initial comments for you.

[Electronic-Waste]

AFAIK, we've removed our support for 1.27: https://github.com/kubeflow/training-operator/blob/1dfa40c12516fc9eb2ce12c5ef52da7d46670457/.github/workflows/unittests.yaml#L21

[andreyvelich]

Good point, let me update it.

[Electronic-Waste]

I think, it will be better if we could provide a standalone installation guide:)

[andreyvelich]

Actually, I am planning to refactor our manifests given that the Cluster Training Runtime needs to be installed after the manager.
I will soon submit a PR.

[astefanutti]

I would guess installing the cluster training runtime requires the CRDs to be installed first, more than the manager to be deployed. Projects tend to separate the steps to install CRDs first, then the rest of the manifests.

[andreyvelich]

Actually, we also need to install manager before we deploy the CTR, since we perform validation and mutation via webhook.

[Electronic-Waste]

Same as above

[vsoch]

Don't most copy paste a full URL that starts with https?

[andreyvelich]

Actually, this is how you can use remote URL with kubectl and kustomize (e.g. they accept the SSH url: github.com:kubeflow/training-operator.git )

[vsoch]

Suggested change

	Ensure that you have access to a Kubernetes clusters with the Kubeflow Training
	Ensure that you have access to a Kubernetes cluster with the Kubeflow Training

[vsoch]

Suggested change

	[the installation guide](/docs/components/training/admin-guides/installation) to quickly deploy
	Kubeflow Training on a local Kind cluster.
	[the installation guide](/docs/components/training/admin-guides/installation) to deploy
	Kubeflow Training.

No reason it needs to be kind, and if there are webhooks it won't be that quick :P

[andreyvelich]

I intentionally add this message to tell that it is super easy to deploy Kubernetes locally, and quickly try Kubeflow Training. I don't want to scare our ML users with "Kubernetes" dependency.
@kubeflow/wg-training-leads @vsoch @astefanutti @franciscojavierarceo @StefanoFioravanzo Any thoughts on how we can phrase it better in docs ?

[astefanutti]

I agree with @vsoch suggestion, it doesn't seem there is a need to be too specific here, "quickly deploy Kubeflow Trainer" straight makes it even less scary here and do not exclude non-Kind options.

[andreyvelich]

How do you think we can make this message better, especially for those users who don't know what is Kubernetes?

[andreyvelich]

@vsoch @astefanutti ?

[vsoch]

Suggested change

	You can chose between installing the latest stable release of the development version from
	You can chose between installing the latest stable release or the development version from

[vsoch]

Suggested change

	The Kubeflow Training is a Kubernetes-native project for large language models (LLMs) fine-tuning
	The Kubeflow Training project is a Kubernetes-native project for large language models (LLMs) fine-tuning

[vsoch]

Ditto (and above this)

[vsoch]

Ditto

[vsoch]

Ditto

[vsoch]

Ditto

[andreyvelich]

We discussed with @thesuperzapper how to structure the new docs, and he suggested that we put new user guides under
/docs/components/training/user-guides-v2
For example: https://deploy-preview-3958--competent-brattain-de2d6d.netlify.app/docs/components/training/user-guides-v2/pytorch/

That will allow us to redirect the V1 docs (e.g. TFJob) to the correct location.

What do you think about it @kubeflow/wg-training-leads ?

[thesuperzapper]

@andreyvelich make sure you also redirect the "index" pages e.g.

• /docs/components/training/explanation/ -> /docs/components/training/legacy-v1/explanation/
• /docs/components/training/user-guides/ -> /docs/components/training/legacy-v1/user-guides/
• /docs/components/training/reference/ -> /docs/components/training/legacy-v1/reference/

Also, I am not sure if it matters, but all other redirects use a trailing / slash.

[andreyvelich]

Good catch, let me fix that.

[franciscojavierarceo]

What do you think about calling it "Kubeflow Trainer"? 🤔

[andreyvelich]

Hmm, this could be one of the options to name this project Kubeflow Trainer/KFTrainer.
@franciscojavierarceo Please can you explain why do you prefer this project name over the KFTraining ?

[franciscojavierarceo]

It would follow HuggingFace's and Lightning.ai's, "trainer" convention: https://huggingface.co/docs/transformers/en/main_classes/trainer
https://lightning.ai/docs/pytorch/stable/common/trainer.html

[andreyvelich]

So we have these:
Project Name:
Kubeflow Trainer

CRD Names:

• TrainJob
• TrainingRuntime

SDK APIs:

from kubeflow.trainer import TrainerClient, Trainer

TrainerClient().train(
    Trainer(
        func=train_func,
    ),
    num_nodes=5,
    runtime_ref="torch-distributed"
)

# For LLMs
from kubeflow.trainer import TrainerClient, Trainer, FineTuningConfig, LoraConfig
TrainerClient().train(
    Trainer(
        fine_tuning_config=FineTuningConfig(
            peft_config=LoraConfig(
                r=4,
            )
    ),
    num_nodes=5, 
    runtime_ref="llama-3.2-8b",
)

What do we think about these names ?

[andreyvelich]

cc @astefanutti @kubeflow/wg-training-leads @Electronic-Waste @deepanker13 @saileshd1402 @seanlaii @kannon92

[chasecadet]

@andreyvelich I don't like that Kubeflow platform combines the products together? Tell me more! I love the idea of bringing components together but also love the idea of installing them separately. If I remember correctly, there is someone dying on that hill :-). That being said, we still don't have conformance.. we still don't have a unified definition of Kubeflow..so it might be a bit early to rename anything tbh. It feels like procrastination chores when we've got big fish to fry here.

Also,

For example, probably we don't want to rename Kubeflow Pipelines to Kubeflow Pipelines Service.

Why not? What are you optimizing for and based on what demand? Have we opened this up to the greater community? Who are our "customers" so to speak and how would they want APIs/Components labeled? This seems like a big decision to make in a vacuum and thought leadership is service (Kelsey has for sure played this game well). What if we opened it up to the greater community? Posted some options on poles via socials we as a community publish? We can then be more data driven. I bet @StefanoFioravanzo has some opinions!

[terrytangyuan]

One concern to me is that training operator is not limited to training use case but can be expanded to parallel computing (via MPIJob). Similarly, PyTorchJob is not limited to PyTorch trainer but more like PyTorch distributed which offers primitives to and abstractions for parallelism, sharding, and communications.

[andreyvelich]

@andreyvelich I don't like that Kubeflow platform combines the products together?

My points don't imply that Kubeflow products cannot function as standalone applications. However, for users interested in seeing how these individual open-source projects integrate seamlessly, the Kubeflow Platform provides a comprehensive, end-to-end machine learning experience.

Why not? What are you optimizing for and based on what demand? Have we opened this up to the greater community?

I believe it will take our community significantly more time to discuss this thoroughly (approximately 1–2 years given the user base of Kubeflow Pipelines).
In the meantime, we should focus on releasing our current project.

One concern to me is that training operator is not limited to training use case but can be expanded to parallel computing (via MPIJob). Similarly, PyTorchJob is not limited to PyTorch trainer but more like PyTorch distributed which offers primitives to and abstractions for parallelism, sharding, and communications.

That is correct, but right now it is out of scope of supported CRDs (TrainJob and TrainingRuntime). Theoretically, users can leverage TrainingRuntimes and TrainJob for distributed inference with MPI, but it would be better if we create dedicated CRDs for it. Also, our kubeflow Python SDK doesn't support it.

---

All in all, naming this project Kubeflow Trainer will help avoid user confusion, considering the following user journey:

1. Cluster Operators install Kubeflow Trainer controller manager into Kubernetes cluster.
2. Cluster Operators configure the required Training Runtimes for ML users.
3. ML Users use Kubeflow Python SDK to create TrainJob objects and interact with the Kubeflow Trainer APIs:

from kubeflow.trainer import TrainerClient

# Get available runtimes.
TrainerClient().list_runtimes()

# Train my ML model
TrainerClient().train(
    runtime_ref="torch-distributed",
    trainer=Trainer(
        func=train_func,
        func_args={"lr": 0.01},
        num_nodes=100,
        resources_per_node={"gpu": 5},
    ),
)

We can always revisit the project name in the future if users tell us that this experience is bad.
What do you think ?

[Electronic-Waste]

------------- Kubeflow Platform -------------
- Kubeflow Workspaces          <---- AI Model Development
- Kubeflow Spark               <---- AI Data Processing
- Kubeflow Trainer             <--- AI Model Training
- Kubeflow Optimizer           <--- AI Model Optimization
- Kubeflow Model Registry      <---- AI Model Management
- Kubeflow Pipelines           <--- Run ML pipelines using the above tools
------------- Kubernetes --------------------

I would support this naming convention, it would be more clear to users than training-operator and katib.

Also cc👀 @Doris-xm @truc0

[tenzen-y]

I'm ok with Kubeflow Trainer. It's challenging to find the name that can cover the whole of the use cases (ML Training / HPC Computing) and all use personals (ClusterOperators / ML Engineers / Researchers / Backend Engineers). I believe that Trainer can flexibly cover all.

But I agree with here discussions since Trainer does not exactly cover the whole of things. Everyone talks about the project name based on different points of view (personas and workload specifications)

Note that Previously, we were seeking the Kubeflow Batch (or Job) as an alternative project name.
However, we declined to introduce the Batch as a name since we believe the Trainer can imply MachineLearning semantics rather than Batch.

[pdarshane]

Should there be a comma between "template" and "check"?

[pdarshane]

I see inconsistent use of "This document". "This guide", "This page" throughout this PR. We can just start the sentence with something like "To contribute to the Kubeflow Training project... " or something similar. Just a suggestion.

[andreyvelich]

I think, historically we've been using this in the beginning of the page:

This document describes how to ....

However, some pages have various messages.
@pdarshane From your point of view, how should we start our guides ?
cc @StefanoFioravanzo @varodrig

[varodrig]

@andreyvelich based on previous conversation, we will not be having a specific page for the contribution/community.

[andreyvelich]

What do you mean @varodrig ? Even right now, individual Kubeflow projects have their own contributor guides:

• Kubeflow Training Operator: https://github.com/kubeflow/training-operator/blob/master/CONTRIBUTING.md
• Kubeflow Katib: https://github.com/kubeflow/katib/blob/master/CONTRIBUTING.md
• Spark Operator: https://www.kubeflow.org/docs/components/spark-operator/developer-guide/
  I thought, we've just discussed whether we should cross-link these guides from the Kubeflow website.

[varodrig]

I created an issue to reflect the conversation we had and made a few updates, feel free to make any suggestions
#3971
but the main idea is to not have individual pages on each project on the website, but continue one centralized place on the website and links to the git repos.

[pdarshane]

We can eliminate "you can" to reduce wordiness.

[pdarshane]

Style guides typically recommend against using "please".

[varodrig]

Additionally, we need to:

• Update Katib documentation to point to v2
• Update Contributions page https://www.kubeflow.org/docs/about/contributing/
• Update Community Page https://www.kubeflow.org/docs/about/community/

[varodrig]

this is under discussion, given that other components do not have this content on the website.we want to ensure this is consistent across the website.

[varodrig]

I created an issue to reflect the conversation we had and made a few updates, feel free to make any suggestions
#3971
but the main idea is to not have individual pages on each project on the website, but continue one centralized place on the website and links to the git repos.

[varodrig]

this is under discussion, given that other components do not have this content on the website. we want to ensure this is consistent across the website.

[varodrig]

what about
"This doc is in progress"

or just remove the section until is fully ready

[andreyvelich]

I will add the getting started example once we finish this PR with @astefanutti: kubeflow/training-operator#2387

[varodrig]

my two cents here is that given that the component's name changed should we said V2? or just Kubeflow Trainer

[andreyvelich]

Maybe we could just say:
Follow this guide for migrating to the new Kubeflow Trainer project.
WDYT @varodrig @kubeflow/wg-training-leads ?

[varodrig]

I recommend to do something like this Legacy Kubeflow Training Operator v1 (since the name changed just Legacy it's maybe not enough for users that are looking for the Kubeflow Training Operator docs ), so keeping the name it'll be helpful

[varodrig]

We also need to update the images on this page that's making a reference to the Training Operator such as the Kubeflow Ecocystem and Kubeflow Components in the ML Lifecycle

[andreyvelich]

Yes, good point. I will update these images.

[varodrig]

Do we need to update the Training Operator Python SDK link to manage Training Operator jobs using Python APIs. in the Kubeflow APIs and SDKs?

[andreyvelich]

@varodrig Where do you want to update it ?

[varodrig]

@andreyvelich it's on the https://www.kubeflow.org/docs/started/architecture/#kubeflow-apis-and-sdks
let me know if this helps

[andreyvelich]

Good catch, let me update it!

[varodrig]

one of the logos in the diagram is broken

[andreyvelich]

@varodrig Please can you point to the diagram which is broken ?

[varodrig]

I just noticed the image is fine but when showing on the web page is not. So, one of the main problems is that the background is black, when loading on the page, because the background is white all the white content from personas titles, arrows, logos titles are not showing. the kubeflow python SDK is the one that is not showing the whole image properly.

.

[andreyvelich]

Interesting, for me the rendering works correct.

[andreyvelich]

Which browser do you use @varodrig ?
@thesuperzapper Do you know what is the right way to export images from drawio to avoid such problems ?

[varodrig]

I'm using Chrome

[tenzen-y]

I have same problems with @varodrig and I am using chrome as well

[andreyvelich]

@sftim @Arhell Do you know why are we getting this issue with Hugo ?
Does Kubernetes also use drawio for the diagrams ?

[sftim]

Kubernetes docs mostly don't use draw.io (some do, there are lots of diagrams). See https://kubernetes.io/docs/contribute/style/diagram-guide/ for what Kubernetes recommends.

[varodrig]

in installing the control plane,
when says : "if you have already installed kubeflow platform" is going to the latest version of kubeflow. Should we replace the link with the 1.9 release?

https://v1-9-branch.kubeflow.org/docs/started/installing-kubeflow/

[andreyvelich]

Actually, the latest version of Kubeflow Platform 1.10 will also include Training Operator v1.

[varodrig]

In the Next Steps section , the link is associated with the latest getting started guide instead of v1

Run your first Training Operator Job by following the Getting Started guide.

[saileshd1402]

LGTM! Just had a few minor doubts

[saileshd1402]

Currently when running this command, I see that the V1 SDK is installed. Should we wait for it to be updated to add it here, or is it fine keep it?

[andreyvelich]

Yes, good point!
Let me remove it until we release SDK to PyPI and keep only command that installs SDK from the source.

[saileshd1402]

Do we need to be worried about links to examples/repo in documentation since we are moving V1 to a separate branch? For example: line 23, line 29 of this file. This would apply to most example job docs AFAIK

[andreyvelich]

Oh, it is a great find! I think, we should link the examples from the release-1.9 branch.
Let me update it.

[astefanutti]

Maybe that line can be removed as it's specific to MacOS and at that stage we can assume the user has installed Kind or have a Kubernetes cluster already.

[astefanutti]

Suggested change

	control plane installed. If it is not set up yet, followÍ
	control plane installed. If it is not set up yet, follow

[astefanutti]

Suggested change

	Kubeflow Trainer on your local Kind cluster.
	Kubeflow Trainer.

It may be just better straight "quickly deploy Kubeflow Trainer". The provided link gives the specifics.

[Electronic-Waste]

Just a few comments:)

[Electronic-Waste]

Should we rename it to Trainer Python SDK since we don't have kubeflow/sdk now?

[andreyvelich]

Actually, we want to rename the kubeflow_training SDK to kubeflow SDK: https://github.com/kubeflow/training-operator/blob/master/sdk_v2/pyproject.toml#L7.
We will not push it to PyPI yet, until we finalize the proposal of creation a new kubeflow/sdk repo.
Thus, I prefer we call it Kubeflow SDK in the docs.