This is the official code repository for the NAACL 2022 paper QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization by Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong.
In our paper, we conduct an extensive comparison of the components of QA-based metrics for factual consistency evaluation in summarization. Our optimized metric builds on QAEval with question consistency filtering and an improved answer overlap metric, leading to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark.
5/2/2022 - Initial commit! :)
You can install qafacteval via pip:
pip install qafactevalYou can also install from source:
git clone https://github.com/salesforce/QAFactEval
cd QAFactEval
pip install -e .Download the required pretrained models using download_models.sh.
See run.py for an example of using the QAFactEval metric:
from qafacteval import QAFactEval
kwargs = {"cuda_device": 0, "use_lerc_quip": True, \
"verbose": True, "generation_batch_size": 32, \
"answering_batch_size": 32, "lerc_batch_size": 8}
model_folder = "" # path to models downloaded with download_models.sh
metric = QAFactEval(
lerc_quip_path=f"{model_folder}/quip-512-mocha",
generation_model_path=f"{model_folder}/generation/model.tar.gz",
answering_model_dir=f"{model_folder}/answering",
lerc_model_path=f"{model_folder}/lerc/model.tar.gz",
lerc_pretrained_model_path=f"{model_folder}/lerc/pretraining.tar.gz",
**kwargs
)
results = metric.score_batch_qafacteval(["This is a source document"], [["This is a summary."]], return_qa_pairs=True)
score = results[0][0]['qa-eval']['lerc_quip']When referencing this repository, please cite this paper:
@misc{fabbri-etal-2022-qafacteval,
title = {QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization},
author = {Alexander R. Fabbri and Chien-Sheng Wu and Wenhao Liu and Caiming Xiong},
year={2022},
eprint={2112.08542},
archivePrefix={arXiv},
primaryClass={cs.CL},
url = {https://arxiv.org/abs/2112.08542},
}This repository is released under the BSD-3 License.