Tools

How do you select LLMs to deploy on a GPU cluster for hundreds of engineers?
LLM benchmark results are proudly published with every new model release — but what’s really behind those numbers?

This talk guides you through designing benchmarks that truly measure the real-world problem-solving capabilities of LLMs.

We’ll explore methods for evaluating business-relevant LLM skills such as:

coding performance,

tool usage and API calling,

behavior in agentic workflows.

After uncovering common flaws in popular benchmarks, we’ll demonstrate alternative techniques for LLM evaluation — including:

ground truth-based testing,

LLM-as-a-judge, and

execution-based validation (e.g., testing LLM-generated code).

We’ll also highlight common pitfalls, such as how prompt variations and generation parameters can subtly and silently skew benchmark results.

Finally, we’ll provide a practical decision framework to help you choose the right LLM, balancing latency, cost, and output quality.

By the end of this talk, you’ll be equipped to go beyond the hype and evaluate LLMs like an engineer — not just by the score.