Load text file for RoBERTa pre-training.

[chiyuzhang94]

I migrate my question from huggingface/transformers#4009 (comment)

I tried to train a Roberta from scratch using transformers. But I got OOM issues with loading a large text file.
According to the suggestion from @thomwolf , I tried to implement datasets to load my text file. This test.txt is a simple sample where each line is a sentence.

from datasets import load_dataset
dataset = load_dataset('text', data_files='test.txt',cache_dir="./")
dataset.set_format(type='torch',columns=["text"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
next(iter(dataloader))

But dataload cannot yield sample and error is:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-12-388aca337e2f> in <module>
----> 1 next(iter(dataloader))

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    361 
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    401     def _next_data(self):
    402         index = self._next_index()  # may raise StopIteration
--> 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    404         if self._pin_memory:
    405             data = _utils.pin_memory.pin_memory(data)

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

KeyError: 0

dataset.set_format(type='torch',columns=["text"]) returns a log says:

Set __getitem__(key) output type to torch for ['text'] columns (when key is int or slice) and don't output other (un-formatted) columns.

I noticed the dataset is DatasetDict({'train': Dataset(features: {'text': Value(dtype='string', id=None)}, num_rows: 44)}).
Each sample can be accessed by dataset["train"]["text"] instead of dataset["text"].

Could you please give me any suggestions on how to modify this code to load the text file?

Versions:
Python version 3.7.3
PyTorch version 1.6.0
TensorFlow version 2.3.0
datasets version: 1.0.1

[lhoestq]

Could you try

load_dataset('text', data_files='test.txt',cache_dir="./", split="train")

?

load_dataset returns a dictionary by default, like {"train": your_dataset}

[chiyuzhang94]

Hi @lhoestq
Thanks for your suggestion.

I tried

dataset = load_dataset('text', data_files='test.txt',cache_dir="./", split="train")
print(dataset)
dataset.set_format(type='torch',columns=["text"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
next(iter(dataloader))

But it still doesn't work and got error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-388aca337e2f> in <module>
----> 1 next(iter(dataloader))

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    361 
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    401     def _next_data(self):
    402         index = self._next_index()  # may raise StopIteration
--> 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    404         if self._pin_memory:
    405             data = _utils.pin_memory.pin_memory(data)

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in __getitem__(self, key)
   1069             format_columns=self._format_columns,
   1070             output_all_columns=self._output_all_columns,
-> 1071             format_kwargs=self._format_kwargs,
   1072         )
   1073 

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1056                 format_columns=format_columns,
   1057                 output_all_columns=output_all_columns,
-> 1058                 format_kwargs=format_kwargs,
   1059             )
   1060         return outputs

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in _convert_outputs(self, outputs, format_type, format_columns, output_all_columns, format_kwargs)
    872                     continue
    873                 if format_columns is None or k in format_columns:
--> 874                     v = map_nested(command, v, **map_nested_kwargs)
    875                 output_dict[k] = v
    876         return output_dict

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    214     # Singleton
    215     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 216         return function(data_struct)
    217 
    218     disable_tqdm = bool(logger.getEffectiveLevel() > INFO)

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in command(x)
    833                     if x.dtype == np.object:  # pytorch tensors cannot be instantied from an array of objects
    834                         return [map_nested(command, i, **map_nested_kwargs) for i in x]
--> 835                 return torch.tensor(x, **format_kwargs)
    836 
    837         elif format_type == "tensorflow":

TypeError: new(): invalid data type 'str'

I found type can be ['numpy', 'torch', 'tensorflow', 'pandas'] only, how can I deal with the string type?

[thomwolf]

You need to tokenize the string inputs to convert them in integers before you can feed them to a pytorch dataloader.

You can read the quicktour of the datasets or the transformers libraries to know more about that:

• transformers: https://huggingface.co/transformers/quicktour.html
• dataset: https://huggingface.co/docs/datasets/quicktour.html

[sipah00]

Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files

mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_

2. Pass paths of small data files to load_dataset

files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')

(On passing the whole dataset file (11GB) directly to load_dataset was resulting into RAM issue)

3. Tokenization

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])

Now you can pass dataset to Trainer or pytorch DataLoader

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))

Hope this helps

[chiyuzhang94]

Thanks, @thomwolf and @sipah00 ,

I tried to implement your suggestions in my scripts.
Now, I am facing some connection time-out error. I am using my local file, I have no idea why the module request s3 database.

The log is:

Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    raise err
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    timeout=timeout
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    chunked=chunked,
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/text/text.py (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection obj
ect at 0x7fff401e0e48>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Traceback (most recent call last):
  File "/scratch/roberta_emohash/run_language_modeling.py", line 1019, in <module>
    main()
  File "/scratch/roberta_emohash/run_language_modeling.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 117, in HG_Datasets
    dataset = load_dataset('text', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/evn_py36/datasets/datasets/src/datasets/load.py", line 590, in load_dataset
    self._validate_conn(conn)
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fff401e0da0>: Failed to establish a new connection: [Errno 110] Connection timed out

Do you have any experience on this issue?

[sipah00]

No, I didn't encounter this problem, it seems to me a network problem

[chiyuzhang94]

I noticed this is because I use a cloud server where does not provide for connections from our standard compute nodes to outside resources.

For the datasets package, it seems that if the loading script is not already cached in the library it will attempt to connect to an AWS resource to download the dataset loading script.

I am wondering why the package works in this way. Do you have any suggestions to solve this issue?

[chiyuzhang94]

I solved the above issue by downloading text.py manually and passing the path to the load_dataset function.

Now, I have a new issue with the Read-only file system.

The error is:

I0916 22:14:38.453380 140737353971520 filelock.py:274] Lock 140734268996072 acquired on /scratch/chiyuzh/roberta/text.py.lock
Found main folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text
Creating specific version folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014
I0916 22:14:38.530371 140737353971520 filelock.py:318] Lock 140734268996072 released on /scratch/chiyuzh/roberta/text.py.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 1019, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 117, in HG_Datasets
    dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 590, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 385, in prepare_module
    os.makedirs(hash_folder_path)
  File "/project/chiyuzh/evn_py36/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014'

I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
I also tried change the environment variables to the writable directory:
export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/
export HF_DATASETS_CACHE=/project/chiyuzh/evn_py36/datasets/cache_dir/

In my scripts, I also changed to:
dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
data_cache_dir = $TMPDIR/data/ that also a writable directory.

But it still try to make directory at /home/chiyuzh/.cache/huggingface/modules/.
Do you have any idea about this issue? @thomwolf

[shizhediao]

Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files

mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_

1. Pass paths of small data files to load_dataset

files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')

(On passing the whole dataset file (11GB) directly to load_dataset was resulting into RAM issue)

1. Tokenization

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])

Now you can pass dataset to Trainer or pytorch DataLoader

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))

Hope this helps

When I run 'dataset = dataset.map(encode, batched=True)',
I encountered a problem like this:

Testing the mapped function outputs
Traceback (most recent call last):
File "", line 1, in
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
for k, dataset in self.items()
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in
for k, dataset in self.items()
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
update_data = does_function_return_dict(test_inputs, test_indices)
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
File "", line 3, in encode
TypeError: init() takes 1 positional argument but 2 were given

[chiyuzhang94]

Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files

mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_

1. Pass paths of small data files to load_dataset

files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')

(On passing the whole dataset file (11GB) directly to load_dataset was resulting into RAM issue)

1. Tokenization

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])

Now you can pass dataset to Trainer or pytorch DataLoader

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))

Hope this helps

When I run 'dataset = dataset.map(encode, batched=True)',
I encountered a problem like this:

Testing the mapped function outputs
Traceback (most recent call last):
File "", line 1, in
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
for k, dataset in self.items()
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in
for k, dataset in self.items()
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
update_data = does_function_return_dict(test_inputs, test_indices)
File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
File "", line 3, in encode
TypeError: init() takes 1 positional argument but 2 were given

What is your encoder function?

[shizhediao]

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

It is the same as suggested:

def encode(examples):
return tokenizer(examples['text'], truncation=True, padding='max_length')

[chiyuzhang94]

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

It is the same as suggested:

def encode(examples):
return tokenizer(examples['text'], truncation=True, padding='max_length')

Do you use this function in a class object?

init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

[shizhediao]

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

It is the same as suggested:

def encode(examples):
return tokenizer(examples['text'], truncation=True, padding='max_length')

Do you use this function in a class object?

init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

Thanks for your reply.
Could you provide some simple example here?
Currently, I do not use this function in a class object.
I think you are right and I was wondering how to construct this class.
I try to modify it based on transformers' LineByLineTextDataset. Am I correct?

class LineByLineTextDataset(Dataset):
"""
This will be superseded by a framework-agnostic approach
soon.
"""

def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
    assert os.path.isfile(file_path), f"Input file path {file_path} not found"
    # Here, we do not cache the features, operating under the assumption
    # that we will soon use fast multithreaded tokenizers from the
    # `tokenizers` repo everywhere =)
    #logger.info("Creating features from dataset file at %s", file_path)
    #with open(file_path, encoding="utf-8") as f:
    #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
    #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)

import glob
files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files)
    batch_encoding= dataset.map(encode, batched=True)
    self.examples = batch_encoding["input_ids"]


def encode(examples):
    return tokenizer(examples['text'], truncation=True, padding='max_length')

def __len__(self):
    return len(self.examples)

def __getitem__(self, i) -> torch.Tensor:
    return torch.tensor(self.examples[i], dtype=torch.long)

[chiyuzhang94]

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

It is the same as suggested:

def encode(examples):
return tokenizer(examples['text'], truncation=True, padding='max_length')

Do you use this function in a class object?
init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

Thanks for your reply.
Could you provide some simple example here?
Currently, I do not use this function in a class object.
I think you are right and I was wondering how to construct this class.
I try to modify it based on transformers' LineByLineTextDataset. Am I correct?

class LineByLineTextDataset(Dataset):
"""
This will be superseded by a framework-agnostic approach
soon.
"""

def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
    assert os.path.isfile(file_path), f"Input file path {file_path} not found"
    # Here, we do not cache the features, operating under the assumption
    # that we will soon use fast multithreaded tokenizers from the
    # `tokenizers` repo everywhere =)
    #logger.info("Creating features from dataset file at %s", file_path)
    #with open(file_path, encoding="utf-8") as f:
    #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
    #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)

import glob
files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files)
    batch_encoding= dataset.map(encode, batched=True)
    self.examples = batch_encoding["input_ids"]


def encode(examples):
    return tokenizer(examples['text'], truncation=True, padding='max_length')

def __len__(self):
    return len(self.examples)

def __getitem__(self, i) -> torch.Tensor:
    return torch.tensor(self.examples[i], dtype=torch.long)

I am also struggling with this adaptation.
I am not sure whether I am right.

I think you don't need to construct class LazyLineByLineTextDataset(Dataset) at all.
torch.utils.data.Dataset is a generator.

Now, we use dataset = dataset.map(encode, batched=True) as a generator. So we just pass dataset to torch.utils.data.DataLoader.