Prepare The Pile for use with T5X

[slippylolo]

Description

We are gonna be using The Pile as our pre-training dataset. We need to get The Pile into SeqIO to be able to use properly with T5X, and pre-process/cache it for pre-training.

Action items

• Download The Pile somewhere accessible to our TPUs/pre-processing machine;
• Create a SeqIO dataset for The Pile;
• Create two options for pre-processing: for causal language modelling and for masked language modelling with denoising (like in T5);
• Run the pre-processing and caching (Adam Roberts can help on this).

Note: we may not need The Pile in its entirety. We are targeting runs ~30GT, so probably that ~50-100GT is fine if it's easier/faster to do.

[craffel]

Download The Pile somewhere accessible to our TPUs/pre-processing machine;

It should probably be in the bigscience GCS bucket.

Create a SeqIO dataset for The Pile;
Create two options for pre-processing: for causal language modelling and for masked language modelling with denoising (like in T5);

See https://github.com/google-research/text-to-text-transfer-transformer/blob/main/t5/data/tasks.py#L45 and https://github.com/google-research/text-to-text-transfer-transformer/blob/main/t5/data/tasks.py#L66 - you should only need to alter the source.

[adarob]

I haven't run SeqIO caching on public Beam myself, but here is an example of someone who has: google/seqio#109

It would be great to improve the SeqIO documentation if similar issues are hit.

[thomasw21]

Closing as we'll be using c4 instead. (Thanks @adarob for preprocessing it for us). Will re-open if we want to switch back.