General-purpose LLM benchmarks miss the mark, because domain-specific enterprise tasks reshuffle the leaderboard

The researchers took Stanford's HELM evaluation framework and bolted on 27 enterprise-focused benchmarks that HELM had never covered. The datasets span four domains: finance (10 English datasets plus 2 Japanese), legal (7 datasets), climate (3 datasets), and cybersecurity (6 datasets). Tasks include classification, named entity recognition, question answering, summarisation, and translation. Every dataset is publicly available and contains at least 100 labelled test cases.

They then ran 8 open-source models through the lot, all under 70 billion parameters. The roster includes Llama 3.1 (8B and 70B), Phi 3.5 (3.8B), Mistral 7B, Granite 3 (8B), Flan UL2 (20B), and two Japanese-specialised models. All models used identical prompts, identical few-shot examples, greedy decoding at temperature zero, and a fixed random seed. No chain-of-thought prompting, no system prompts, no tricks.

The headline result is that no single model dominates across enterprise domains. Llama 3.1-70B tops the finance benchmark on 7 of 10 tasks, scoring 0.874 Weighted F1 on news headline classification and 0.802 Entity F1 on the challenging FiNER-139 numerical NER task. In cybersecurity it leads on 4 of 6 tasks, reaching 0.896 F1 on threat intelligence mapping. So far, so predictable for the largest model in the set.

The story gets interesting when you compare domain performance to general benchmarks. Flan UL2 ranks first on IMDb sentiment classification (0.975 accuracy) and first on CNN-DailyMail summarisation (0.299 Rouge-L). Move to domain-specific sentiment in finance or legal, and it drops to sixth and fifth respectively. The reason is telling: financial and legal texts express positive or negative sentiment through domain idiom ("year-over-year decline in operating margin") rather than the straightforward polarity words ("great", "terrible") that movie reviews favour.

Smaller models occasionally punch above their weight. Granite 3 (8B), trained on finance and legal data, holds the top summarisation score in both BillSum (0.312 Rouge-L) and Legal Summarisation (0.271 Rouge-L), beating Llama 3.1-70B at roughly nine times the parameter count. Mistral 7B leads on legal judgement prediction (0.845 Weighted F1). In climate classification, Phi 3.5 at only 3.8B parameters tops all models on wildfire tweet classification at 0.796 Weighted F1.

The authors attribute the rank shuffles primarily to training data composition. Flan UL2, pre-trained on the C4 corpus (filtered Common Crawl), was published in early 2023, before the push to include domain-specific corpora became standard. Its training data almost certainly lacked the volume of financial, legal, and cybersecurity text found in more recent datasets. Granite 3, by contrast, explicitly includes FDIC filings, finance textbooks, and EDGAR documents in its training pipeline, which explains its stable performance across both finance and legal domains despite having only 8 billion parameters.

The NER results reveal a separate problem. Traditional BIO-tag sequence labelling simply fails with decoder-only LLMs. The team adopted an extraction-based approach (the model outputs "New York (location)" rather than a tag sequence), and found it needs 20+ few-shot examples to work properly. Even then, numerical NER (extracting KPIs from SEC filings, say) remains "a particularly challenging task" where the 70B model manages only 0.697 Adjusted F1 on KPI-Edgar. Specialised domain vocabulary is not just about sentiment, it extends to structural conventions in how numbers, entities, and relations appear in enterprise text.

Practitioners choosing an LLM for an enterprise application face a concrete problem: the model that tops MMLU or IMDb might rank fifth or sixth on the task they actually care about. This paper provides the benchmarks, prompts, and HELM integration code to run that comparison directly. The entire framework is open-sourced and being merged into the main HELM repository.

The Japanese results add a useful data point for multilingual enterprise deployment. Granite 8B Japanese outperformed Llama 3 ELYZA JP 8B across all four Japanese finance tasks, including a BLEU score of 0.123 on English-to-Japanese financial translation. That is still a low number in absolute terms, and it signals how far specialised multilingual enterprise NLP has to go.