Explaining Large Language Model Decisions Using Shapley Values
2024-12-28
Large language models (LLMs) offer exciting possibilities for simulating human behavior, but their decision-making processes lack transparency. This paper introduces a novel approach based on Shapley values to interpret LLM behavior and quantify the contribution of each prompt component to the model's output. Through two applications, the study reveals that LLM decisions are susceptible to "token noise," where the model disproportionately reacts to tokens with minimal informative content. This raises concerns about the robustness and generalizability of insights from LLMs in simulating human behavior, highlighting the need for careful prompt engineering and a nuanced understanding of their limitations when used in research.