Added example notebook for translation with ct2 model. #262
Conversation
uahmed93
force-pushed
the
uahmed/ct2_translation_example_notebook
branch
2 times, most recently
from
1b7e864 to
5ecd4fe
Compare
Signed-off-by: Ahmed Umair <[email protected]>
uahmed93
force-pushed
the
uahmed/ct2_translation_example_notebook
branch
from
5ecd4fe to
cef1c34
Compare
| "/home/uahmed/Desktop/NeMo-Curator/.ndc/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator LinearRegression from version 1.4.2 when using version 1.5.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n", | ||
| "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n", | ||
| " warnings.warn(\n", | ||
| "GPU: 0, Part: 0: 0%| | 0/10 [05:33<?, ?it/s]1.74s/it]\n", | ||
| "GPU: 0, Part: 0: 100%|██████████| 10/10 [00:17<00:00, 1.77s/it]\n" | ||
| ] | ||
| }, |
There was a problem hiding this comment.
Given its a tutorial, we should show the output that writing it out, maybe try doing both .
| "source": [ | ||
| "import os\n", | ||
| "\n", | ||
| "os.environ[\"DASK_DATAFRAME__QUERY_PLANNING\"] = \"False\"\n", |
There was a problem hiding this comment.
You can remove this, no need to add it now.
There was a problem hiding this comment.
Yep, this should be removed.
| "metadata": {}, | ||
| "source": [ | ||
| "# Indic Translation\n", | ||
| "This notebook demonstrate an example use of nemo-curator for Indic language generation via translation from English language. This workflow is accelarated by CrossFit, a library that leverages intellegent batching and RAPIDS to accelerate the offline inference on large datasets. \n", |
There was a problem hiding this comment.
Would suggest mentioning that we support large scale translation using multi node multi gpu setup
| "# Indic Translation\n", | ||
| "This notebook demonstrate an example use of nemo-curator for Indic language generation via translation from English language. This workflow is accelarated by CrossFit, a library that leverages intellegent batching and RAPIDS to accelerate the offline inference on large datasets. \n", | ||
| "This example uses ctransalte2 model from [here](https://indictrans2-public.objectstore.e2enetworks.net/it2_preprint_ckpts/en-indic-preprint.zip), taken from IndicTrans2 github repo, [here](https://github.com/AI4Bharat/IndicTrans2/tree/main?tab=readme-ov-file#multilingual-translation-models)" | ||
| ] |
There was a problem hiding this comment.
Add a clear demarkation of import section here
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Define output directory\n" |
There was a problem hiding this comment.
| "## Define output directory\n" | |
| "##### Define output directory\n" |
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", |
There was a problem hiding this comment.
Add a table of contents here.
See https://github.com/NVIDIA/NeMo-Curator/blob/cef1c3408684e9ce96e8125f041378407daf0a7b/tutorials/image-curation/image-curation.ipynb example for input
Signed-off-by: Ahmed Umair <[email protected]>
|
Hi @VibhuJawa / @sarahyurick , |
| "source": [ | ||
| "import os\n", | ||
| "\n", | ||
| "os.environ[\"DASK_DATAFRAME__QUERY_PLANNING\"] = \"False\"\n", |
There was a problem hiding this comment.
Yep, this should be removed.
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "@dataclass\n", |
There was a problem hiding this comment.
How about removing this entire cell and creating a new script in https://github.com/NVIDIA/NeMo-Curator/tree/main/nemo_curator/classifiers?
There was a problem hiding this comment.
We cant really have another classifier just yet, we have to have this as a tutorial as we dont want to support this globally just yet. We can probably create a utils file here to make this cleaner though.
There was a problem hiding this comment.
Ah ok I see, using a utils file sounds good to me.
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "class IndicTranslation(DistributedDataClassifier):\n", |
There was a problem hiding this comment.
Same comment as above, why don't we move all of this logic into https://github.com/NVIDIA/NeMo-Curator/tree/main/nemo_curator/classifiers?
Description
This PR is adding a translation example to hindi languge via ct2 model.
Issue here
This example depends on CrossFit's PR
Checklist