Bug fix: Only append eod token once when packing / tokenizing

[le1nux]

What does this PR do?

Some HF tokenisers such as xlm-roberta-large add special tokens (e.g., eod token) automatically when encoding text, whereas others, such as gpt2, do not add special tokens.

This side-effect in the transformers library has lead to the eod token being appended twice when tokenizing / packing our data. We added a check for this and only append the eod token once now:

modalities/src/modalities/dataloader/create_packed_data.py

Lines 327 to 330 in 1c1ccdc

	if not token_byte_string.endswith(self._encoded_eos_token_as_bytes):
	token_byte_string = token_byte_string + self._encoded_eos_token_as_bytes
	return token_byte_string

Additionally, we now enforce now that the eod token is a special token.

Additionally, I added a script that verifies the consistency of the indexation and tokenization of a given JSONL file. We run the indexation and tokenization routines in modalities and compare it to tokenized JSONL file to which we applied the HF tokenizer directly.

Checklist before submitting final PR

• My PR is minimal and addresses one issue in isolation
• I have merged the latest version of the target branch into this feature branch
• I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
• I have run a sample config for model training
• I have checked that all tests run through (python tests/tests.py)
• I have updated the internal changelog (CHANGELOG_DEV.md)

[mali-git]

LGTM! Minor comments.

[flxst]

Great work! :)

Found some minor issues and left some comments.

[fromm-m]

LGTM