AI prototypes are easy to demo and surprisingly hard to trust. A copilot gives a convincing answer, an AI agent completes a workflow once, and a prompt looks better after a few tweaks – but how can a team know whether the system actually improved? How can it catch regressions, measure quality, control cost, and decide whether a new version is ready for production?

In this practical session, Eugene Sergueev shows a step-by-step evaluation playbook for large language model (LLM) applications and AI agents. The talk focuses on moving from demo-driven AI development to a repeatable engineering process with real-world cases, quality rubrics, baselines, evaluation runs, thresholds, release gates, and production feedback loops.

The session walks through a lightweight reference architecture that teams can adopt without building a large evaluation platform first. Eugene explains how to collect 20–50 representative real-world cases, define what “good” and “bad” mean, create a golden dataset, compare prompt, model, and agent versions, and detect regressions before they reach users.

The talk also covers where rule-based checks, human review, and LLM-as-a-judge approaches work well – and where they fail. Eugene discusses how teams can use shadow runs, A/B tests, expert corrections, and production feedback to improve AI systems in small, measurable steps.

Attendees will leave with a practical framework they can reuse in their own teams: an evaluation loop, a quality rubric structure, a release-readiness checklist, and a reference architecture for evaluating LLM-powered workflows and AI agents.

The session is designed for engineers, AI practitioners, product engineering leaders, and platform teams who have already seen impressive AI demos and now need a way to make AI systems reliable, measurable, and safe enough for real production use.

This is not a vendor or platform demo. The session focuses on practical engineering patterns that can be applied across different AI stacks: internal copilots, LLM-powered product features, document question-answering (QA) systems, AI-assisted development workflows, and tool-using agents.

The talk is structured as a step-by-step implementation guide and includes:

• a lightweight evaluation loop for LLM apps and agents;
• an example structure for a golden dataset;
• a quality rubric for AI outputs;
• examples of metrics and thresholds;
• a release-readiness checklist;
• common anti-patterns when teams evaluate AI manually;
• guidance on where human review, automated checks, and LLM-based graders should be used.