Conquering Dumb LLM Search Judges with Classic ML

The author explores using a local LLM as a search relevance judge, a cost-effective alternative to OpenAI. Individual LLM judgments are unreliable, so the article proposes combining multiple LLMs' assessments of various product attributes (name, classification, description, etc.) using traditional machine learning (e.g., decision trees) to improve accuracy. Experiments show this approach can predict human preferences and reveal the logic behind human labels, aiding search engine optimization.
Read more