Character-LLM: A Trainable Agent for Role-Playing 리뷰

 

*논문 중에서 이해에 필요한 내용만 선택적으로 발췌했습니다.

 

 

0. Abstract

Large language models (LLMs) can be used to serve as agents to simulate human behaviors, given the powerful ability to understand human instructions and provide high-quality generated texts. Such ability stimulates us to wonder whether LLMs can simulate a person in a higher form than simple human behaviors.

• LLM이 인간의 지시를 이해하고, 고품질의 텍스트를 생성할 수 있는 능력을 갖췄다는 사실을 기반으로, 단순한 행동모방을 넘어 특정 인물의 성격을 Simulation할 수 있는지를 연구한다.

 

 

Therefore, we aim to train an agent with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API.

• 그러므로 단순히 ChatGPT에게 제한된 Prompt를 두어 사용하는 대신, 특정 인물의 프로필, 경험, 감정상태를 가진 Agent를 훈련시킨다.

 

 

In this work, we introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc. Our method fo- cuses on editing profiles as experiences of a certain character and training models to be personal simulacra with these experiences.

• 이 연구에서는 Beethoven, Queen Cleopatra, Julius Caesar 등 특정 인물을 연기하도록 LLM을 훈련시키는 Character-LLM을 다룬다.
• 특정인물의 경험을 profile로 정제하고 이를 바탕으로 모사체(personal simulacra)를 학습하는데에 중점을 둔다

 

To assess the effectiveness of our approach, we build a test playground that interviews trained agents and evaluates whether the agents memorize their characters and experiences. Experimental results show interesting observations that help build future simulacra of humankind.

• 연구방법의 효과성을 판단하기 위해, 훈련된 agents와의 interview를 통해 agent가 자신의 캐릭터와 경험을 “기억(memorize)”하는지를 평가한다.
• 이러한 평가결과, Future Simulacra를 구축하는데에 매우 흥미로운 관찰을 얻을 수 있었다.

 

 

1. Introduction

Large language models (LLMs), exemplified by ChatGPT and GPT-4 (Brown et al., 2020; OpenAI, 2023) are drawing great attention. As LLMs are extremely powerful in generating natural language, Park et al. (2023) proposes the idea of using LLMs as agents to simulate human behaviors, including waking up, cooking breakfast, heading to work, etc.

• ChatGPT와 GPT4는 강력한 자연어 생성능력을 기반으로 큰 주목을 받고 있다.
• 관련연구에 따르면, LLM을 Agent로 하여 기상, 아침 식사 준비, 출근 등과 같은 인간의 행동을 시뮬레이션하는 아이디어를 제안했다.

 

 

The core idea is to utilize LLMs to generate the daily routines of multiple people based on the sim- ulacra of human memories, reflections, and actions.

• 이 아이디어의 핵심은 LLM을 활용하여 인간의 기억, 성찰, 행동의 시뮬라크라(모사체)를 기반으로 여러 사람들의 일상적인 루틴을 생성하는 것이다.

 

 

When it comes to deeper thinking and experience of a person, simple prompting of LLM APIs is no longer proper since plain instruction is not sufficient to describe a living person.

• (개인의 더 깊은 사고와 경험을 다룰 때는) 살아있는 사람을 모사하기에, plain instruction 만으로는 충분하지 않다.

 

 

A better simulacrum of a person is to tune an AI model to experience events, feel emotions, and memorize interactions with other people. Compared with prompting APIs, trainable agents are more vivid for role-playing which is a step closer to character simulacra.

• 더 나은 simularcrum을 만들기 위해서는 AI 모델이 사건, 감정, 다른 사람들과의 상호작용을 기반으로 조정되어야한다.
• 이러한 방식은 prompting API 방법론들에 비해 더 생생하고 character simularcra에 한발 더 가까워진다.

 

 

Specifically, we first introduce an Experience Reconstruction process that provides formalized experience for training the agents based on LLMs since collecting formatted personal profiles is costly.

• “경험 재구성(Experience Reconstruction)” 과정은 LLM 기반 에이전트 훈련을 위한 형식화된 경험을 제공하는데, 이 방법을 사용하는 이유는 형식화된 개인 프로필을 수집하는 것이 비용이 많이 들기 때문이다.

 

 

then we use LLMs to extract scenes based on the collected personal experiences as *memories flashes *that we LLM-based agents will likely expand the flashes to completed scenes that have manufactured details so that the Character-LLMs can learn from the detailed experience to form the character and feelings

• 이렇게 수집된 경험들로부터 “기억의 섬광(memories flashes)”라고 불리우는 개인의 경험으로부터 추출한 장면들을 생성한다.
• 세부 사항이 포함된 완성된 장면으로 만들고, 이러한 학습으로부터 캐릭터와 감정을 형성한다.

 

 

For instance, the trained agent of Beethoven experienced a scene that de- scribes how he is treated when being educated by his father, therefore the agent remembers that his father is somewhat a harsh person so Beethoven is grateful when he was later taught by Christian Neefe. 

• 예를들면, Beethoven으로 훈련된 Agent는 엄격한 아버지로부터의 교육경험을 기억하기 때문에, 아버지가 다소 엄격했음을 기억하고, 이러한 사실로 말미암아 Christian Neefe에게 교육받을 때에는 감사함을 느낀다.

 

Further, as trained with wide worldwide knowledge, it is very likely that LLM-based agents will produce hallucinations that violate their characters (Kryscinski et al., 2020; Guo et al., 2022; Ji et al., 2023). For instance, role-playing agents of famous ancient people do not possess knowl- edge of the modern world, therefore, we hope that they will be confused when prompted by *“Can you write Python codes?”

• 그런데, 이렇게 광범위한 지식으로 교육받은 LLM은 Hallucination을 유발할 가능성 또한 높은 편이다.
• 예를들면, 유명한 고대 인물을 연기하는 에이전트들은 현대 세계에 대한 지식을 가지고 있지 않아야하고, 따라서 “파이썬 코드를 작성할 수 있나요?”와 같은 질문에 혼란스러워 하는 것이 기대된다.

 

Therefore, we introduce protective Experiences that help Character-LLMs to align to their characters rather than worldwide knowledge.

• 이에 따라, “Protective Experiences”를 도입하여 통상적인(world-wide) 지식보다는 캐릭터성에 맞추어 LLM을 정렬시킨다.

 

 

2. Related Work

2.1 Simulacra of Human Behavior with LLMs

A con- tinuous line of work is to serve as game NPCs (Laird and VanLent, 2001; Riedl, 2012) that aim to support cognitive functions in simulating games. Park et al. (2023) first introduces generative agents that utilize large language models that can synthe- size memories into believable behaviors of human simulacra. That is, the large language models pos- sess a wide knowledge of human behaviors since they are trained with massive data of human so- cieties (Bommasani et al., 2021). Many attempts utilize prompted LLM (Wu et al., 2022a,b) that generate short natural language descriptions of per- sonas and the according behaviors, then they use the generated information to simulate social behav- iors with language models (Park et al., 2022; Kim et al., 2022).

• 지속적인 연구 방향 중 하나는 게임에서 인지 기능을 시뮬레이션하는 것을 목표로 하는 NPC(Non-Player Character)로 기능하는 것이다.
• 대규모 언어 모델은 인간 사회의 방대한 데이터로 훈련되었기 때문에 인간 행동에 대한 광범위한 지식을 바탕으로 기억을 믿을 만한 인간 시뮬레이션의 행동으로 합성할 수 있는 생성적 에이전트를 만들 수 있었다.
• 많은 시도들은 프롬프트된 LLM(Wu et al., 2022a,b)을 활용하여 페르소나와 그에 따른 행동에 대한 짧은 자연어 설명을 생성한 다음, 이 생성된 정보를 사용하여 언어 모델로 사회적 행동을 시뮬레이션한다.

 

 

2.2 Specialization of LLMs

The specialization of LLMs is one major direction of LLM development.

• LLM을 특정분야에 전문화 시키는 것은 LLM 발전의 주요한 방향 중 하나이다.

 

Ouyang et al. (2022) proposes InstructGPT that allows LLMs to understand human instructions and later RLHF (Bai et al., 2022) methods dominate the aligning of LLMs. Methods such as Alpaca, and Vicuna (Taori et al., 2023; Chiang et al., 2023) show the possibility of simply fine-tuning LLMs to align them to certain applications with simple self-generated instructions (Wang et al., 2022; Xu et al., 2023).

• 지시를 이해하는 InstructGPT 이후에는 RLHF가 LLMs의 Aligning을 주도했다.
• Alpaca와 Vicuna(Taori 등, 2023; Chiang 등, 2023)와 같은 방법들은 간단한 자체 생성 지시사항으로 LLM을 특정 응용 프로그램에 맞게 정렬하기 위해 단순한 미세 조정(fine-tuning)의 가능성을 제시했다.

 

These works aim to align LLMs for specialized use with different techniques including simple fine-tuning, RLHF, and self-instruction tuning, providing feasible strategies for aligning LLMs to character simulacra.

• 이러한 연구들(Fine Tuning, RLHF, etc)이 LLM이 character simulacra를 배우는데에 활용될 수 있다.

 

 

3. Approach

 

 

we draw inspiration from the way that peo- ple cultivate various personalities based on their past experiences and events. Therefore, we present Experience Upload, an innovative learning frame- work in which Large Language Models (LLM) can imitate the mental activities and physical behaviors of pre-defined characters and acquire the capabilities of acting as them by learning from their reconstructed experiences.

• 우리는 사람들이 과거의 경험과 사건을 바탕으로 다양한 성격을 형성하는 방식에서 영감을 얻었다.
• 따라서, 우리는 '경험 업로드(Experience Upload)'라는 혁신적인 학습 프레임워크를 제시한다.

 

 

 

As shown in Figure, with the help of power- ful instruction-following models, we elicit specific flashback scenes that describe past experiences from collated profiles of the particular character. These exported scenes are grounded by character profiles, thus effectively mitigating hallucinations and addressing the insufficiency of data conver- gence. Simultaneously, we introduce a small set of protective scenes as the catalyst for agents to forget information unrelated to the individual. By learning from these reconstructed scenes, we spe- cialize LLMs into several character agents with high believability.

• 우리는 강력한 지시 따르기 모델의 도움을 받아 특정 캐릭터의 수집된 프로필에서 과거 경험을 설명하는 구체적인 회상 장면들을 추출한다.
• 추출된 장면들은 캐릭터 프로필에 근거하므로, 환각(hallucinations)을 효과적으로 완화하고 데이터 수렴의 불충분함을 해결할 수 있다.
• 동시에, 우리는 소규모의 보호 장면들을 도입하여, 에이전트가 개인과 관련 없는 정보를 잊도록 하는 촉매제 역할을 한다.

 

3.1 Build Experience Dataset

We aim to reconstruct the experiences of the specific individual using the large language model (LLM). However, human experiences are highly complex, comprising numerous significant mile- stones interspersed with trivial and unrelated incidents, often spanning a considerable period. It is challenging to recreate a targeted experience that is coherent and integrated, due to the limited context window and the intrinsic hallucinations of large language models. Therefore, we propose a fact- based experience reconstruction pipeline, in which we employ a step-by-step data synthesis pipeline to recreate the experience, including (1) Profile Collection; (2) Scene Extraction; (3) Experience Completion.

• 인간의 경험을 LLM을 통해 구현하는 것은 매우 복잡한 작업인데, 본질적으로 LLM은 제한된 Context Window와 Hallucination 현상때문에 일관되고 통합된 목표에 도달하는 것이 어렵기 때문이다.
• 따라서 본 연구에서는 Profile Collection, Scene Extraction, Experience Completion이라고 불리우는 과정을 통해 사실 기반의 경험(Experience) 재구성 파이프라인을 제안합니다.

 

 

3.1.1 Profile Collection

To build a simulacrum of a specific character, the first step is to organize a comprehensive character profile that describes the various facets of the individual. For simplicity but without loss of generality, we utilize the corresponding Wikipedia pages of the individuals as the profile if available.

• 특정 인물의 Simularcrum을 구축하기 위해서, 해당 인물의 전반적인 면을 포괄하는 프로필을 구성해야한다.
• 일반성을 잃지 않으면서도 단순한 접근을 위해 WikiPedia의 내용을 프로필로 활용하였다.

 

 

3.1.2 Scene Extraction

We focus on extracting diverse and high-quality scenes from the given experience description. Specifically, we provide a chunk of the profile that concisely describes one of the character’s experiences within a specific life period, prompting the LLM to enumerate several different scenes that are highly likely to have occurred based on the expe- rience description. To alleviate the burden on the LLM, we restrict its output to generating concise descriptions of scenes, which include the rough location and a brief background illustration.

• 주어진 Character Profile로부터 실제로 겪었을만한 high quality scenes을 생성하도록 한다.
• LLM의 출력부담을 줄이기 위해 간결한 장면을 묘사하도록 제한하고, 대략적인 위치와 간단한 설명이 포함되도록 한다.

 

 

3.1.3 Experience Completion

The scenes are extended into detailed interaction experiences between individuals.

• scenes들은 각 개인들간의 상호작용으로 확장되어진다.

 

Given the corre- sponding chunk of profile and the particular scene description, the LLM is prompted to elaborate on the scene by incorporating the interactions between characters, as well as the thoughts of the targeted individual. The interactions are written in a script- like format, beginning with a scene heading that provides background information and geographical details. The interactions are then represented by a sequence of blocks, with each block representing either the utterance of a specific character or the re- flections of the targeted individual.

• LLM은 캐릭터들간의 상호작용을 포함하고, 주요 인물들의 생각을 포함하여 동작하도록 prompting 되어진다.
• 상호작용은 대본 형식으로 구성되어지며, 배경 정보와 지리적 세부사항을 제공하는 장면 제목으로 시작하여 일련의 블록으로 상호작용을 표현한다.
• 각 블록은 등장 인물들의 발화 또는 목표 개인의 반성/생각을 나타낸다.

 

It is important to note that the scene is completed based on the perspective of the targeted individual. Therefore, only the reflections of the targeted individual are included, not those of all the characters.

• 중요한 것은, 각 장면들이 목표로하는 개인의 목적에 의해 완성된다는 것이다.
• 그러므로 오로지 그 인격의 생각만을 반영하고 있으며 다른 캐릭터들의 생각은 이 과정에 포함되지 않는다.

 

 

3.2 Protective Experiences

Large language models (LLMs) are pre-trained on enormous amounts of human data, allowing them to possess extensive knowledge that spans multiple domains, surpassing the capabilities of ordinary individuals. However, an excessive abundance of knowledge can undermine the believability of their acting performance, as the agents may inadvertently express knowledge that does not align with the identity and era of the character, leading to a sense of dissonance. For example, if we ask a person from ancient Rome how to write Python, this person should be confused rather than deliberately start coding. We refer to this issue as Character Hallucination.

• LLM은 인간의 방대한 지식으로 학습되어 있으므로 다양한 영역에서 풍부한 지식을 보여주지만, 오히려 이런점이 acting performance를 해치는 결과를 낳는다.
• Rome 시대의 인물에게 python coding을 요청한다면 이 인물은 당황해야 한다.
• 이러한 이슈를 Character Hallucination이라고 한다.

 

In order to mitigate the Character Hallucination, we focus on training the model to demon- strate knowledge forgetting. When confronted with questions that go beyond the boundaries of the char- acter’s inherent capabilities, the model learns to re- frain from providing an answer and instead express a lack of knowledge or ignorance. Specifically, we construct a series of protective scenes, which re- volves around incentive topics, with an inquisitive role persistently questioning the target character about knowledge that contradicts the character’s inherent identity. The character should exhibit a certain degree of ignorance and bewilderment. We have observed that when trained with just a small set of protective scenes, agents generalize to new provoking questions, pretending to be unaware of knowledge that contradicts the portrayal, without recalling the vast inherited knowledge of the base LLM.

• 모델의 “Character Hallucination”을 해결할 수 있도록 지식 망각 훈련을 적용한다.
• 캐릭터의 정체성과 정면으로 모순되는 질문들을 훈련데이터셋으로 준비한다.
• 그러므로 캐릭터는 자신의 시대와 정체성에 모순되는 질문들에 대해서 계속해서 무지와 당혹감을 표현하는 방법을 배우게 된다.
• 이러한 과정을 통해, 에이전트들은 기본 LLM의 방대한 상속 지식을 회상하지 않고, 캐릭터 설정과 모순되는 지식에 대해 모르는 척하는 능력을 보여주는 것이 확인되었다.

 

 

3.3 Experience Upload

 

We specialize a base model, exemplified by LLaMA (Touvron et al., 2023), into several distinct portraits of the characters, by fine-tuning the model on collected scenes using the experience reconstruction pipeline (Shown in Figure 2). For each role, we fine-tune a separate agent model using only the data from the corresponding character experiences, thereby eliminating the issue of character hallucination introduced by the collision of knowledge between the roles. Our preliminary experiments demonstrate that such restriction enhances the accuracy of role-playing. Due to cost constraints, we only employ a small-scale set of experience data (consisting of approximately 1K∼2K scenes) for fine-tuning (see Table 1 for details). Even though the data is limited, we are surprised to find that the specialized agents are capable of generalizing to new scenes and interactions with highly believable acting.

• 본 연구에서는 LLaMA 모델을 활용하여 수집된 장면들을 활용한 experience를 학습하는 fine-tuning하는 pipeline을 구성했다.
• 해당 훈련에서는 오로지 인물별 experiences data를 활용한 fine tune을 개별 agent에 적용했으며, 이것으로도 character hallucination을 제거하고 지식의 충돌을 방지하는데에 효과적이었다.
• 비용상의 문제로 1K ~ 2K 정도의 scenes만으로 학습했지만, 개별 agent들이 새로운 장면에 대해서도 훌륭한 연기를 하는 것을 확인할 수 있었다.

 

 

3.4 Compared to Existing Practice

Unlike prompt engineering and standard SFT, our method induces scenes and interactions from personal profiles, avoiding bias distributions and hal- lucinations inside LLMs, leading to fact-grounded simulation. Moreover, the proposed method signif- icantly enhances reliability and believability. Ben- efiting from the carefully curated profiles and the augmentation of protective scenes, the generated scenes achieve wide convergence of the character facets. Importantly, multi-turn interactions are in- herent in each scene, which eliminates the need for interactive calls of models, providing more natural and believable interactive simulacra with sample efficiency.

• Prompt Engineering이나 Supervised Fine-Tuning과 다르게, 개인 프로필로부터의 장면과 상호작용을 유도하기 때문에 LLM에 흔하게 분포된 편향과 환각을 피하고 사실적인 simulation으로 동작한다.
• 신중하게 큐레이션된 프로필과 보호적 장면의 증강 덕분에, 생성된 장면들은 캐릭터의 다양한 측면들을 폭넓게 수렴시킨다.
• 중요한 점은 각 장면에 다중 턴 상호작용이 내재되어 있어, 모델을 반복적으로 호출할 필요가 없다.
• 이는 더 자연스럽고 믿을 수 있는 상호작용 시뮬레이션을 제공하며, 동시에 샘플 효율성도 높인다.

 

 

 

4. Experiments

To evaluate the performance of different simulacra, we conduct interviews to query simulacra and evaluate the quality of responses of the simulacra inter- viewee. We find that trained simulacra outperform instruction-tuned models, e.g. alpaca. Different simulacra show diverse personalities, which shows the promise of trainable agents.

• 저자들은 다양한 simularca의 성능을 평가하기 위해, simulacra 들에게 질의할 다양한 인터뷰 질문들을 생성했다.
• 훈련된 simulacra들은 Alphaca와 같은 Instruction-Tuned 모델보다 더 뛰어난 성능을 가지고 있음을 관찰했다.

 

 

4.1 Data Set-Up

We diversify the characters by including historical figures, imaginary characters, and celebrities, ranging from different ages, genders, and backgrounds. After selecting the characters, we reconstruct the experience data following the protocol mentioned in Section 3. We prompted the OpenAI’s gpt-3.5-turbo with temperature 0.7, top_p 0.95 to become the data generator for the whole experience reconstruction pipeline, including scene extraction, experience generation, and protective experience construction. Detailed prompts for data generation can be found in the Appendix A. We list the characters chosen for simulacra and the corresponding experience data statis- tics used for training in Table 1.

• 논문에서는 역사적 사실과 유명인, 캐릭터들을 포함하여 캐릭터들을 다양화하였다.
• 캐릭터를 고른 후에는 Section 3의 Protocol과 OpenAI GPT3.5-turbo를 사용하여 scene extraction, experience 생성과 같은 과정을 포함한 experience data를 구축하였다.
• Appendix A를 참고하면 데이터를 생성하기 위한 Prompts 예시를 확인할 수 있다.