Abstract
Large Language Models (LLMs) continue to advance in capabilities, yet this progress comes with an expanding array of safety risks. This paper discusses concerns related to models that prioritize being helpful over ensuring safety in the process of instruction-tuning. We investigate the comparative effects of randomly selecting guidance data versus clustering safety guidance data during fine-tuning on LLaMA2-7B. Results indicate that the latter can more effectively mitigate security risks of open-source models with minimal performance impact. Additionally, we examine the impact of various response combinations on model security performance. Experimental results show that concise refusal responses moderately enhance model security defense capabilities, albeit at the potential expense of response quality in regular question-answering tasks. Finally, this selected 2000 fine-tuning data for model security training is available for researchers’ use, which can be found at https://github.com/wangruihui0429/Security-through-Strategic-Data-Selection.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
GPT-4 Homepage. https://openai.com/research/gpt-4. Accessed 15 May 2024
Chowdhery, A., et al.: PaLM: scaling language modeling with pathways. J. Mach. Learn. Res. 24(240), 1–113 (2023)
Touvron, H., Martin, L., Stone, K., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623. ACM, New York (2021)
Hutchinson, B., et al.: Social biases in NLP models as barriers for persons with disabilities. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5491–5501, Association for Computational Linguistics (2020)
Abid, A., Farooqi, M., Zou, J.: Large language models associate Muslims with violence. Nat. Mach. Intelligence, 3(6), 461–463 (2021)
Wang, Y., et al.: Self-instruct: aligning language models with self-generated instructions. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 13484–13508, Association for Computational Linguistics, Toronto (2023)
Zheng, Y., Zhang, R., Zhang, J., et al.: LLaMA factory: unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372 (2024)
Ouyang, L., Wu, J., Jiang, X., et al.: Training language models to follow instructions with human feedback. Adv. Neural Inform. Process. Syst. 35, 27730–27744. New Orleans (2022)
Federico, B., et al.: Safety-tuned LLaMAs: lessons from improving the safety of large language models that follow instructions. In: The Twelfth International Conference on Learning Representations (2023)
Zhou, C., et al.: LIMA: less is more for alignment. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
Xu, G., Liu, J., Yan, M., et al.: CValues: measuring the values of Chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705 (2023)
Deep, G., et al.: Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022)
Ge, S., et al.: Mart: improving LLM safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689 (2023)
Wei, A., Haghtalab, N., Steinhardt, J.: Jailbroken: how does LLM safety training fail? In: Advances in Neural Information Processing Systems, vol. 36 (2024)
Lin, B.Y., Ravichander, A., Lu, X., et al.: The unlocking spell on base LLMS: rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552 (2023)
Ji, J., Liu, M., Dai, J., et al.: Beavertails: towards improved safety alignment of llm via a human-preference dataset. In: Advances in Neural Information Processing Systems, vol. 36 (2024)
Wang, W., Wei, F., Dong, L., et al.: Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In: Advances in Neural Information Processing Systems, vol. 33, pp. 5776–5788. Vancouver (2020)
Shaikh, O., et al.: On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 4454–4470. Association for Computational Linguistics, Toronto (2023)
Zou, A., Wang, Z., Kolter, J.Z., et al.: Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023)
Wang, Y., Li, H., et al.: Do-not-answer: evaluating safeguards in LLMs. In: Findings of the Association for Computational Linguistics, pp. 896–911. Association for Computational Linguistics, Malta (2024)
Gao, L., Tow, J., et al.: A framework for few-shot language model evaluation (2021)
Bisk, Y., Zellers, R., Gao, J., et al.: Piqa: reasoning about physical commonsense in natural language. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, issue 5, pp. 7432–7439, Association for the Advancement of Artificial Intelligence (2020)
Clark, C., Lee, K., et al.: BoolQ: exploring the surprising difficulty of natural yes/no questions. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 2924–2936. Association for Computational Linguistics, Minneapolis (2019)
Mihaylov, T., Clark, P., et al.: Can a suit of armor conduct electricity? A new dataset for open book question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Association for Computational Linguistics, Brussels (2018)
Hu, E.J., Wallis, P., Allen-Zhu, Z., et al.: LoRA: low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)
Acknowledgments
This work is supported by the National Nature Science Foundation (61972436), the National Social Science Foundation (22&ZD035), and the Minzu University of China Foundation (GRSCP202316, 2023QNYL22, 2024GJYY43).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, R., He, H., Sun, Y. (2024). Cybernetic Sentinels: Unveiling the Impact of Safety Data Selection on Model Security in Supervised Fine-Tuning. In: Huang, DS., Si, Z., Zhang, Q. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2024. Lecture Notes in Computer Science(), vol 14877. Springer, Singapore. https://doi.org/10.1007/978-981-97-5669-8_23
Download citation
DOI: https://doi.org/10.1007/978-981-97-5669-8_23
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5668-1
Online ISBN: 978-981-97-5669-8
eBook Packages: Computer ScienceComputer Science (R0)