The talk begins with a company where generative AI (GenAI) is already in production, but nothing is tested. Every change is risky, and nobody can tell whether the system has improved or simply broken something.

Marcin Szymaniuk explains how to bring structure from three perspectives:

• Developer: how to build evaluation loops, datasets, and basic tests for retrieval-augmented generation (RAG), retrieval, and document pipelines
• Team lead/manager: what is realistic to test, what is expensive, and how to plan work around uncertainty
• Business: how to move fast without guessing and how real user feedback is more valuable than any single metric

He then covers how to evaluate non-deterministic systems:

• RAG and document-quality metrics, including relevance, faithfulness, and completeness
• Conversation testing using synthetic scenarios
• Using large language models (LLMs) as judges when rule-based checks fail

Marcin also discusses the trade-offs: not everything should be automated, and not everything needs perfect scoring. Testing GenAI is not about achieving perfect correctness – it is about building enough confidence to ship, iterate, and avoid breaking what already works.