[Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.

[JasonCZH4]

Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.

Steps to reproduce the bug

First

I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict {text:'aaa'}.
Their path are /data/train.json, /data/val.json, /data/test.json
I load dataset with data_files argument:

files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)

And I get:

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})

However, If I load dataset with data_dir argument:

ds = load_dataset(
            path='json',
            data_dir='./data',)

And I get:

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})

Two results are not the same. Their behaviors are not equal, even if the statement here said that their behaviors are equal.

Second

If some filename include 'test' while others do not, load_dataset only return test dataset and others files are abandoned.
Given two files named test.json and 1.json
Each one has a simple dict {text:'aaa'}.
I load the dataset using:

ds = load_dataset(
            path='json',
            data_dir='./data',)

Only test is returned, 1.json is missing:

DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})

Things do not change even I manually set split='train'

Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration, I hope I can manually decide whether to do so. Sometimes users may accidentally put a test string in the filename but they just want a single train dataset. If the number of files in data_dir is huge, it's not easy to find out what cause the second situation metioned above.

Environment info

datasets==3.2.0
Ubuntu18.84

[lhoestq]

Hi ! data_files with a list is equivalent to data_files={"train": data_files} with a train test only.

When no split are specified, they are inferred based on file names, and files with no apparent split are ignored

[JasonCZH4]

Thanks for your reply!
files with no apparent split are ignored. Is there a option that I can choose to ignored it or not as I mention aboved? Thanks!

[lhoestq]

To include all the files, the best way is to pass data_files yourself. There is no option to disable split detection at the moment

[JasonCZH4]

Thanks! I hope you guys can consider adding this option in the future. :)