What is an LLM evaluation harness? A deep dive into lm-eval-harness

An LLM evaluation harness is a tool that fills the gap between fine-tuning a model and having a reproducible, comparable, and defendable evaluation metric. It helps with comparability and regression detection by providing a local leaderboard with the tasks you care about. You need to define the three to five tasks that map to your actual use case, plus one or two general capability anchors. The harness handles the boring-but-critical parts of loading the model, running inference, and scoring it against a ground-truth key.