FlagAI-Open · Anhforth · May 30, 2022 · May 29, 2022 · May 29, 2022 · May 29, 2022
diff --git a/README.md b/README.md
@@ -124,7 +124,8 @@ for text in test_data:
 * [NER with RoBerta-WWM](/docs/TUTORIAL_17_BERT_EXAMPLE_NER.md)
 * [Writing with GPT-2](/docs/TUTORIAL_18_GPT2_WRITING.md)
 * [Title generation with T5](/docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md)
-* [Supported tasks](/docs/TUTORIAL_20_SUPPORTED_TASKS.md)
+
+[//]: # (* [Supported tasks]&#40;/docs/TUTORIAL_20_SUPPORTED_TASKS.md&#41;)
 
 
 This session explains how the base NLP classes work, how you can load pre-trained models to tag your

diff --git a/doc_zh/GLM.md b/doc_zh/GLM.md
@@ -12,6 +12,7 @@
 | Autoencoding    | ✅   | ×         | ×          |
 | Encoder-Decoder | -   | ✅         | -          |
 | GLM             | ✅   | ✅         | ✅          |
+
 GLM的主要功能包括：
 
 - 任务一：文本的一些区间会被屏蔽（参照自动编码的做法）。 这些区间将被随机重新排列，并以自动回归方式进行预测。屏蔽的区间覆盖原始文本的15%。

diff --git a/doc_zh/TUTORIAL_2_DATASET.md b/doc_zh/TUTORIAL_2_DATASET.md
@@ -31,10 +31,10 @@
 分类任务微调有着两种形式：一种是普通的微调，一种是提示学习的方法。提示学习需要为任务额外构建一个完形填空的模板，它更适用于低资源以及小样本的情况。接下来我们以提示学习为例来介绍分类任务中的数据处理方法：
 ### 分类任务应用代码
 ```python
-import torch.utils.data
-from flagai.data.dataset import SuperGlueDataset
+import torch
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import CollateArguments
+from flagai.data.dataset import SuperGlueDataset
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSuperglueStrategy
 
 # 得到默认参数
@@ -133,7 +133,7 @@ FlagAI目前支持自动加载下列分类数据集：
 ```python
 example = dataset[3]  # 数据集里第3个样例
 ```
-比如说，上一步`CommitmentBank`的样例会被处理成如下的形式
+比如说，上一步`CommitmentBank`的样例会被处理成如下的形式:
 
 <div align=center><img src="img/dataset_figure_2.png" width="500px"></div>
 
@@ -345,12 +345,17 @@ class ExamplePVP(PVP):
 ```
 ### 预训练的任务处理实例代码
 ```python
+from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.dataset import BlockDataset
+from flagai.data.dataset.block.data_utils import split_ds, get_dataset_lazy, add_args
+from flagai.test_utils import PretrainDatasetArguments
+
 tokenizer = GLMLargeChTokenizer(add_block_symbols=True,
                                 add_task_mask=True,
                                 add_decoder_mask=False,
                                 fix_command_token=True)
 
-ds_args = DatasetArguments()
+ds_args = PretrainDatasetArguments()
 
 tokenizer = GLMLargeChTokenizer(fix_command_token=True,
                                 add_block_symbols=True,
@@ -402,10 +407,10 @@ datasets = create_dataset(tokenizer, should_split=True)
 ## 数据处理：生成任务微调 
 ### 生成任务应用代码
 ```python 
-import torch.utils.data
+import torch
 from flagai.data.dataset import Seq2SeqDataset
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import Seq2SeqCollateArguments
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSeq2seqStrategy
 
 # 得到默认参数

diff --git a/docs/GLM.md b/docs/GLM.md
@@ -9,12 +9,14 @@ The **GLM model**, proposed in [GLM: General Language Model Pretraining
 with Autoregressive Blank Infilling](https://arxiv.org/abs/2103.10360), is based on a slightly different strategy: autoregressive blank infilling. 
 
 It claims to perform well in the three main categories of NLP taks: classification, unconditional generation, and conditional generation tasks.
+
 | Framwork        | NLU | Cond.Gen. | Uncond.Gen |
 |-----------------|-----|-----------|------------|
 | Augoregressive  | -   | -         | ✅          |
 | Autoencoding    | ✅   | ×         | ×          |
 | Encoder-Decoder | -   | ✅         | -          |
 | GLM             | ✅   | ✅         | ✅          |
+
 The key features of GLM include:
 
 - First task: Several spans of the text are masked following the idea of autoencoding. Those spans will be randomly rearranged and be predicted in an autoregressive manner. The masked spans covers 15% original tokens.

diff --git a/docs/TUTORIAL_2_DATASET.md b/docs/TUTORIAL_2_DATASET.md
@@ -5,14 +5,16 @@ The process of constructing a dataset is the data preprocessing process of NLP.
 
 At present, there are three kinds of data preprocessing in the project, namely, fine-tuning for classification tasks, pre-training, and fine-tuning for generation tasks. We will expand on them separately in the following.
 
-## Data processing: fine-tuning for classification tasks ([prompt-learning mode](TUTORIAL_7_PROMPT_LEARNING.md))
+## Data processing: fine-tuning for classification tasks 
+
+There are two forms of tuning for classification tasks: one is fine-tuning, and the other is the [prompt-tuning](/docs/TUTORIAL_7_PROMPT_LEARNING.md). Prompt-tuning requires an additional cloze template for the task, which is more suitable for limited data. Let's take prompt learning as an example to introduce the data processing method in classification tasks:
 ### Application code
 
 ```python
-import torch.utils.data
-from flagai.data.dataset import SuperGlueDataset
+import torch
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import CollateArguments
+from flagai.data.dataset import SuperGlueDataset
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSuperglueStrategy
 
 # get default parameters
@@ -49,23 +51,34 @@ dataset = SuperGlueDataset(task_name='cb',
                            dataset_type='train',
                            tokenizer=tokenizer)
 ```
-#### 1. Load the dataset
+`SuperGlueDataset`is the function in this step，and its major parameters are introduced below：
+
+`task_name`: Identifier of dataset, Supported datasets and their identifiers are given at the table in [1.Load the dataset](#1.Load the dataset).
+
+`data_dir`: Data will be automatically downloaded to `data_dir` directory, which is `./dataset` by default.
+
+`dataset_type`: It can be train/dev/test, which represents train/validation/test set is going to be preprocessed.
+
+`tokenizer`: Constructed tokenizer as introduced in [Tutorial1](/docs/TUTORIAL_1_TOKENIZER.md).
+
+
 
-After setting `task_name` to the abbreviation of the task name, the relevant data will be automatically downloaded in the background. FlagAI currently supports automatic loading of the following classification datasets:
 
-| dataset name                                 | short name| Language | Benchmark   |
-|----------------------------------------------|----------|------|----------|
-| Broadcoverage Diagnostics                    | boolq    | Eng   | SuperGLUE |
-| CommitmentBank                               | cb       | Eng   | SuperGLUE |
-| Choice of Plausible Alternatives             | copa     | Eng   | SuperGLUE |
-| Multi-Sentence Reading Comprehension         | muiltirc | Eng   | SuperGLUE |
-| Recognizing Textual Entailment               | rte      | Eng   | SuperGLUE |
-| Words in Context                             | wic      | Eng   | SuperGLUE |                                                   
-| The Winograd Schema Challenge                | wsc      | Eng   | SuperGLUE |
-| Ant Financial Question Matching Corpus       | afqmc    | Zh    | CLUE     |
-| Short Text Classificaiton for News           | tnews    | Zh    | CLUE     |
+#### 1.Load the dataset
 
-The dataset will be automatically downloaded to the address corresponding to `data_dir`, which defaults to the `./dataset` directory of the project.
+FlagAI currently supports automatic loading of the following classification datasets:
+
+| dataset name                                 | Identifier | Language | Benchmark   |
+|----------------------------------------------|------------|------|----------|
+| Broadcoverage Diagnostics                    | boolq      | Eng   | SuperGLUE |
+| CommitmentBank                               | cb         | Eng   | SuperGLUE |
+| Choice of Plausible Alternatives             | copa       | Eng   | SuperGLUE |
+| Multi-Sentence Reading Comprehension         | muiltirc   | Eng   | SuperGLUE |
+| Recognizing Textual Entailment               | rte        | Eng   | SuperGLUE |
+| Words in Context                             | wic        | Eng   | SuperGLUE |                                                   
+| The Winograd Schema Challenge                | wsc        | Eng   | SuperGLUE |
+| Ant Financial Question Matching Corpus       | afqmc      | Zh    | CLUE     |
+| Short Text Classificaiton for News           | tnews      | Zh    | CLUE     |
 
 The downloaded dataset directory will contain three files, corresponding to the training set data, the validation set data, and the test set data. Take the CommitmentBank dataset as an example, the `train.jsonl` in the directory corresponds to the training set, `val.jsonl `corresponds to the validation set, `test.jsonl` corresponds to the test set. Generally, the training set and test set contain label information, but the test set does not. These data files will be processed separately in the next process.
 
@@ -98,18 +111,19 @@ In this step, we will unify the data structures of different datasets to facilit
 | meta   | an optional dictionary to store arbitrary meta information                        | dict |
 | ids    | an optional numeric index                                                    | int  |
 
-For example, the example of CommitBank in the previous step will be processed into the following form
+When the dataset is built, you can view one of the samples directly in the code by indexing:
+
+```python
+example = dataset[3]  # The third example in dataset 
+```
+
+For instance, the example of CommitBank in the previous step will be processed into the following form
 
 <div align=center><img src="img/dataset_figure_2.png" width="500px"></div>
 
 Noted that if text_a and text_b cannot be filled with background text information because the data structure is too complex, you can put the rest of the information in meta.
 
-When the dataset is constructed, you can view one of the samples directly in the code by indexing:
-
 
-```python
-example = dataset[3]  # The third example in dataset 
-```
 
 ### Organize the data into input to the model
 
@@ -148,7 +162,7 @@ In the first case, the label categories contained in the dataset are limited. Fo
 
 <sup>1</sup>: seq_length represents the specified maximum length of each input vector
 
-<sup>2</sup>: The process of adding special characters: add the [CLS] symbol at the beginning of the sentence and the [EOS] symbol at the end of the sentence until the length reaches seq_length. If the text output by cloze has two paragraphs, it will be in the middle Add [SEP] symbol
+<sup>2</sup>: As illustrated in the following figure, the process of adding special characters: add the [CLS] symbol at the beginning of the sentence and the [EOS] symbol at the end of the sentence until the length reaches seq_length. If the text output by cloze has two paragraphs, it will be in the middle Add [SEP] symbol
 <div align=center><img src="img/dataset_figure_5.png" width="500px"></div>
 
 <sup>3</sup>: num_labels represents the number of options in the cloze problem

diff --git a/docs/TUTORIAL_7_PROMPT_LERANING.md b/docs/TUTORIAL_7_PROMPT_LERANING.md
@@ -19,32 +19,6 @@ We can design a meaningful cloze template as below
 
 <div align=center><img src="img/prompt_figure_1.png" width="500px"></div>
 
-And we can implement the template through the following code:
-
-```python
-def get_parts(self, example: InputExample):  
-    '''A simplified version of the actual code'''
-    # Construct the template
-    cloze_ids = [
-        example.text_a,' question: is it true, false or neither that',
-         example.text_b, ' answer:', [self.mask]]
-    return cloze_ids
-```
-
-The label in various datasets can have multiple types including boolean, integer and string. We need to implement a verbalizer that maps the labels to the text string of choices in cloze question. For example, here the 'entailment', 'contradiction' and 'neutral' labels can be mapped to 'true' ,'false' and 'neither', and each time the answer to be filled is always selected among those choices. 
-
-Verbalizer is essentially a mapping relationship, and we can define it in a Python dictionary as shown below:
-
-```python
-VERBALIZER = {
-    "contradiction": [" false"],
-    "entailment": [" true"],
-    "neutral": [" neither"]
-}
-def verbalize(self, label) -> List[str]:
-    return CbPVP.VERBALIZER[label]
-```
-
 ### Prefix Prompts
 Instead of designing a prompt in real natural language, prefix prompts directly modifies the embedding space. It inserts a sequence of task-specific vectors and freezes the LM parameters. An advantage of prefix prompts is that embeddings of templates are no longer limited by parameters in pre-trained LMs. 
 
@@ -54,10 +28,10 @@ Here is an example that compares prefix tuning and fine-tuning in the embedding
 
 
 ## Reference
-[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)
+Liu, P. (2021, July 28). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting. . . arXiv.Org. https://arxiv.org/abs/2107.13586 
 
-[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998)
+Ding, N. (2021, November 3). OpenPrompt: An Open-source Framework for Prompt-learning. arXiv.Org. https://arxiv.org/abs/2111.01998
 
-[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676)
+Schick, T. (2020, January 21). Exploiting Cloze Questions for Few Shot Text Classification and. . . arXiv.Org. https://arxiv.org/abs/2001.07676
 
-[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190)
+Li, X. L. (2021, January 1). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv.Org. https://arxiv.org/abs/2101.00190
diff --git a/examples/glm_pretrain/train.py b/examples/glm_pretrain/train.py
@@ -1,24 +1,12 @@
+
 from flagai.data.tokenizer import GLMLargeChTokenizer
 from flagai.model.glm_model import GLMForSeq2Seq
 from flagai.trainer import Trainer
 from flagai.data.dataset import ConstructBlockStrategy
 from flagai.data.dataset import BlockDataset
 from flagai.data.dataset.block.data_utils import split_ds, get_dataset_lazy, add_args
 from flagai.data.dataset.superglue.control import DEFAULT_METRICS
-
-
-class DatasetArguments:
-
-    def __init__(self):
-        self.task_mask = True  # Distinguished the generation and gap-sentence mask
-        self.block_mask_prob = 0.1
-        self.block_lm = True  # Whether do masking
-        self.masked_lm = False  # Whether do simple masking (same symbol among masks)
-
-        self.pre_tokenize = True
-        self.no_lazy_loader = True
-        self.half_lazy_loader = False
-        self.sentinel_token = False
+from flagai.test_utils import PretrainDatasetArguments
 
 
 if __name__ == '__main__':
@@ -36,7 +24,7 @@ def __init__(self):
 
     model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch')
 
-    ds_args = DatasetArguments()
+    ds_args = PretrainDatasetArguments()
 
     tokenizer = GLMLargeChTokenizer()
 
@@ -70,35 +58,9 @@ def create_dataset(tokenizer, should_split):
         collate_fn = ConstructBlockStrategy(
             tokenizer, 512, eod_token=tokenizer.get_command('eos').Id)
     metric_methods = DEFAULT_METRICS['pretrain']
-    # trainer.train(model,
-    #               collate_fn=collate_fn,
-    #               train_dataset=datasets[0],
-    #               valid_dataset=datasets[1],
-    #               metric_methods=metric_methods)
-    import torch
-
-    train_loader = torch.utils.data.DataLoader(datasets[0],
-                                               batch_size=16,
-                                               shuffle=False,
-                                               num_workers=1,
-                                               drop_last=False,
-                                               pin_memory=False,
-                                               collate_fn=collate_fn)
-    for data_iterator in train_loader:
-        dct = data_iterator
-        '''
-        input_ids torch.Size([16, 2, 256])
-        labels torch.Size([16])
-        position_ids torch.Size([16, 2, 2, 256])
-        attention_mask torch.Size([16, 2])
-        target_ids torch.Size([16, 2, 256])
-        logit_mask torch.Size([16, 2, 256])
-        loss_mask torch.Size([16, 2])
+    trainer.train(model,
+                  collate_fn=collate_fn,
+                  train_dataset=datasets[0],
+                  valid_dataset=datasets[1],
+                  metric_methods=metric_methods)
 
-        '''
-        for key, value in dct.items():
-            try:
-                print(key, value.size())
-            except:
-                print(key, len(value))
-        break
diff --git a/examples/glm_seq2seq/train.py b/examples/glm_seq2seq/train.py
@@ -22,7 +22,7 @@ def test_init_trainer_pytorch(self):
                           eval_interval=5,
                           log_interval=50,
                           experiment_name='glm_large',
-                          pytorch_device='cpu',
+                          pytorch_device='cuda',
                           load_dir=None,
                           lr=1e-4)
         print("downloading...")