Token classification using Phobert Models for 🇻🇳Vietnamese

Get started in seconds with verified environments. Run script below for install all dependencies

bash ./install_dependencies.sh

The input data's format of 🍜VPhoBertTagger follows VLSP-2016 format with four columns separated by a tab character, including of word, pos, chunk, and named entity. Each word which was segmented has been put on a separate line and there is an empty line after each sentence. For details, see sample data in 'datasets/samples' directory. The table below describes an example Vietnamese sentence in dataset.

The dataset must put on directory with structure as below.

├── data_dir
|  └── train.txt
|  └── dev.txt
|  └── test.txt

The commands below fine-tune PhoBert for Token-classification task. Models download automatically from the latest Hugging Face release

python main.py train --task vlsp2016 --run_test --data_dir ./datasets/vlsp2016 --model_name_or_path vinai/phobert-base --model_arch softmax --output_dir outputs --max_seq_length 256 --train_batch_size 32 --eval_batch_size 32 --learning_rate 3e-5 --epochs 20 --early_stop 2 --overwrite_data

or

bash ./train.sh

Arguments:

• type (str,*required): What is process type to be run. Must in [train, test, predict, demo].
• task (str, *optional): Training task selected in the list: [vlsp2016, vlsp2018_l1, vlsp2018_l2, vlsp2018_join]. Default: vlsp2016
• data_dir (Union[str, os.PathLike], *required): The input data dir. Should contain the .csv files (or other data files) for the task.
• overwrite_data (bool, *optional) : Whether not to overwirte splitted dataset. Default=False
• load_weights (Union[str, os.PathLike], *optional): Path of pretrained file.
• model_name_or_path (str, *required): Pre-trained model selected in the list: [vinai/phobert-base, vinai/phobert-large,...] Default=vinai/phobert-base
• model_arch (str, *required): Punctuation prediction model architecture selected in the list: [softmax, crf, lstm_crf].
• output_dir (Union[str, os.PathLike], *required): The output directory where the model predictions and checkpoints will be written.
• max_seq_length (int, *optional): The maximum total input sequence length after WordPiece tokenization. Sequences longer than this will be truncated, and sequences shorter than this will be padded. Default=190.
• train_batch_size (int, *optional): Total batch size for training. Default=32.
• eval_batch_size (int, *optional): Total batch size for eval. Default=32.
• learning_rate (float, *optional): The initial learning rate for Adam. Default=1e-4.
• classifier_learning_rate (float, *optional): The initial classifier learning rate for Adam. Default=5e-4.
• epochs (float, *optional): Total number of training epochs to perform. Default=100.0.
• weight_decay (float, *optional): Weight deay if we apply some. Default=0.01.
• adam_epsilon (float, *optional): Epsilon for Adam optimizer. Default=5e-8.
• max_grad_norm (float, *optional): Max gradient norm. Default=1.0.
• early_stop (float, *optional): Number of early stop step. Default=10.0.
• no_cuda (bool, *optional): Whether not to use CUDA when available. Default=False.
• run_test (bool, *optional): Whether not to run evaluate best model on test set after train. Default=False.
• seed (bool, *optional): Random seed for initialization. Default=42.
• num_workers (int, *optional): how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. Default=0.
• save_step (int, *optional): The number of steps in the model will be saved. Default=10000.
• gradient_accumulation_steps (int, *optional): Number of updates steps to accumulate before performing a backward/update pass. Default=1.

The command below start Tensorboard help you follow fine-tune process.

tensorboard --logdir runs --host 0.0.0.0 --port=6006

All experiments were performed on an RTX 3090 with 24GB VRAM, and a CPU Xeon® E5-2678 v3 with 64GB RAM, both of which are available for rent on vast.ai. The pretrained-model used for comparison are available on HuggingFace.

[1] Kenton, J. D. M. W. C., & Toutanova, L. K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT (pp. 4171-4186).

python main.py predict --model_path outputs/best_model.pt

python main.py demo --model_path outputs/best_model.pt