Knowledge base
Measuring the effect of AI training: a before-and-after measurement
By Bob Biesemans · published on
AI training costs time and money. How do you measure whether staff really work differently afterwards? A practical approach with a baseline, a follow-up and a fair comparison.
Short answer
Measure the effect of AI training with a baseline and a follow-up on tasks from real work, using different tasks at the same level the second time. Compare per skill and per department, not per person. A satisfaction survey measures how the training was received, not what changed.
Satisfaction is not an effect
After training, most organisations ask whether it was useful. That measures how the training was received, not what changed. Whether people now check AI answers or stop pasting customer data into a free chatbot, you only know by measuring it.
The four steps
1. Baseline: have staff carry out tasks from their own work, such as checking an AI summary against its source or getting AI to draft a customer email. Note per skill where it goes wrong.
2. Targeted training: aim the training at what went wrong in the baseline, per department. General AI explanations for everyone are rarely the best spend.
3. Follow-up: measure again a few weeks later with different tasks at the same level. Repeating the same questions mostly measures memory.
4. Compare: look per skill and per department at what improved, what stayed the same and what still needs attention.
Comparing fairly
Give everyone in a group the same kind of tasks, so you compare results and not people. Look at the pattern per department rather than ranking individuals. And measure everyone the same way, whatever their role or experience.
What you gain
You see which training pays off and which doesn't, you can put the budget where the gap is largest, and you have a documented before-and-after that is strong evidence of the measures you took under Article 4 of the AI Act.
Frequently asked questions
How long after training should I measure again?
After a few weeks, so you see what stuck in daily work and not only what is still fresh.
Can I reuse the same test?
Better not. The same questions mostly measure memory. Use different tasks at the same level.
Does a before-and-after measurement count for Article 4?
Yes, a documented before-and-after is strong evidence that you took measures and that they had an effect.