This repository contains the official release of the model "BanglaBERT" and associated downstream finetuning code and datasets introduced in the paper titled "BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla" accpeted in Findings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: NAACL 2022.

• BanglaBERT
  • Table of Contents
  • Models
  • Datasets
  • Setup
  • Training & Evaluation
  • Benchmarks
  • Acknowledgements
  • License
  • Citation

The pretrained model checkpoint is available here. To use this model for the supported downstream tasks in this repository see Training & Evaluation.

Note: This model was pretrained using a specific normalization pipeline available here. All finetuning scripts in this repository uses this normalization by default. If you need to adapt the pretrained model for a different task make sure the text units are normalized using this pipeline before tokenizing to get best results. A basic example is available at the model page.

We are also releasing the Bangla Natural Language Inference (NLI) and Bangla Question Answering (QA) datasets introduced in the paper.

• NLI
• QA

For installing the necessary requirements, use the following snippet

$ git clone https://github.com/csebuetnlp/banglabert
$ cd banglabert/
$ conda create python==3.7.9 pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch -p ./env
$ conda activate ./env # or source activate ./env (for older versions of anaconda)
$ bash setup.sh 

• Use the newly created environment for running the scripts in this repository.

To use the pretrained model for finetuning / inference on different downstream tasks see the following section:

• Sequence Classification.
  • For single sequence classification such as
    • Document classification
    • Sentiment classification
    • Emotion classification etc.
  • For double sequence classification such as
    • Natural Language Inference (NLI)
    • Paraphrase detection etc.

• Token Classification.
  • For token tagging / classification tasks such as
    • Named Entity Recognition (NER)
    • Parts of Speech Tagging (PoS) etc.
• Question Answering.
  • For tasks such as,
    • Extractive Question Answering
    • Open-domain Question Answering

• Zero-shot cross-lingual transfer-learning

• Supervised fine-tuning

The benchmarking datasets are as follows:

• SC: Sentiment Classification
• NER: Named Entity Recognition
• NLI: Natural Language Inference
• QA: Question Answering

We would like to thank Intelligent Machines and Google TFRC Program for providing cloud support for pretraining the models.

Contents of this repository are restricted to non-commercial research purposes only under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

If you use any of the datasets, models or code modules, please cite the following paper:

@inproceedings{bhattacharjee-etal-2022-banglabert,
    title     = {BanglaBERT: Lagnuage Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla},
    author = "Bhattacharjee, Abhik  and
      Hasan, Tahmid  and
      Mubasshir, Kazi  and
      Islam, Md. Saiful  and
      Uddin, Wasi Ahmad  and
      Iqbal, Anindya  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
      booktitle = "Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022",
      month = july,
    year      = {2022},
    url       = {https://arxiv.org/abs/2101.00204},
    eprinttype = {arXiv},
    eprint    = {2101.00204}
}