[NLP] LLaMa 2

Last wednesday (July 19th), Meta has released another version of its own Large Language Model - LLaMa 2. This model ranges from 7 billion and 70 billion parameters. There are fine-tuned LLMs, called LLaMa-2 Chats, with same range of parameters, that are optimized for chatting. This article is to summarize the paper and describe noteworthy changes that were demonstrated in this model.

Pre-Training

For the pre-training, unlike the former model, LLaMa2 model does not show explicit sources of texts that are used as training.

This model has manipulated pre-training approach of using optimized auto-regressive transformer. Indeed, instead of directly applying such a method, there were processes like data cleaning, increased contexted lengths, more tokens, and implementation of grouped-query attention (GQA) to improve the inference scalability.

The table above demonstrates the feature differences of LLaMa and LLaMa2 models. Note that the context length doubled, GQA has been applied in larger models, and the number of tokens increased generally for the new model.

Fine-Tuning

Fine-tuning was done mainly on chat-optimized models with instruction tuning and reinforcement learning with human feedback (RLHF).

For supervised fine-tuning, although supervising data are available from third-party, it was necessary to utilize high quality prompts for fine-tuning. There were more than 27,000 annotations for the prompt engineering.

Another technique was RLHF - to further align model behavior with human preferences and information following. Along with standard RLHF algorithm known as Proximal Policy Optimization, there was another algorithm applied known as rejection sampling fine-tuning. This is the process of sampling $K$ outputs from the model then selecting the best candidate.

These two algorithms were applied separately first then combined. After the five series of fine-tuning, the helpfulness increased according to the diagram above.

Discussions

One of the concerns for large language models recently is the safety. For the fields that may be controversial and violate the circumstances, LLaMa 2 tried hindering the potential threats for the thriving of the model.

These graphs demonstrate how LLaMa 2 performs regarding safety violations for adversarial prompts. This shows how the model has been trained to focus on the safety issue.

These two diagrams are for helpfulness of the answers provided. The models with different parameters outperform or perform-at-similar-level compared with other models according to the diagram.

The overall paper can be found here.