How robust is your assessment centre? Seven warning signs of limited validity

Imagine you are reviewing the results of an assessment centre for a leadership position. The assessors have compared their observations, the candidates have received scores against your competency model, and the report gives you a clear recommendation. You have invested time and expertise in the process, and you are about to use its conclusions to make an important decision.

If a colleague asked why the recommended candidate is better suited to the role, you could point to the scores. Explaining how those scores were established, and why they provide a sound basis for that decision, takes a closer look.

The answer depends on choices that are easy to miss in the finished report: how requirements were established, which behaviour the exercises elicited, what assessors knew beforehand, and how they interpreted the rating scales.

Research on personnel selection shows why these details deserve attention. Lievens and Sackett (2017) describe how the way a procedure presents tasks and evaluates responses can influence its quality. For assessment centres, design and delivery need to work together. A well-designed exercise still depends on how it is conducted and scored.

The following seven questions help you evaluate the quality of your own assessment centre, from its design to the evidence behind its results.

1. Has your competency model been checked against the target role?

A corporate competency model can provide a useful language across recruitment, development and performance management. Before using it to assess candidates, you still need to establish what its competencies mean for the target role.

Take “strategic thinking”. In one position, it may involve anticipating changes in a market. In another, it may mean translating a business strategy into operational priorities. The label alone tells you little about the situations a candidate needs to handle or what a good response would look like.

A practical requirements analysis starts with the work. Interviews or workshops with knowledgeable role holders and managers can identify critical situations, difficult decisions and examples of effective and ineffective behaviour. These can then be used to prioritise requirements, specify the relevant competencies and choose assessment tasks. Future changes in the role should be considered alongside current demands.

This follows the logic set out in SIOP’s guidance on personnel selection: the analysis of work establishes the link between job requirements and assessment content. An existing competency model can support that work if it is sufficiently detailed and well grounded (SIOP, 2018).

The warning sign is a corporate model adopted without a clear account of its relevance to the role. Ask for the connection between actual work situations, the requirements being assessed and the evidence candidates are expected to provide.

2. How many competencies must an assessor judge at once?

More competencies can make an assessment look more comprehensive. Each additional rating also creates work for the assessor.

Imagine following several people in a group exercise while taking notes and distinguishing between communication, influence, collaboration, empathy, leadership and conflict management. Some behaviours may be relevant to several of these dimensions. Others may be difficult to classify at all.

Kleinmann and Ingold (2019) describe the combination of fast-moving interactions, limited observation time and multiple rating demands as a central challenge of assessment centre work. A detailed form does not remove that challenge.

The warning sign is a rating assignment whose complexity has not been tested in practice. Can assessors distinguish the criteria, capture sufficient evidence and complete their ratings in the available time? Or do similar explanations end up supporting almost every score?

Review the workload per assessor and exercise. Consider the number of candidates, the overlap between competencies, the rating materials and the time available afterwards. A pilot can show where the assignment needs simplifying or redistributing.

There is no single maximum that suits every design. What matters is whether the assessment produces meaningful distinctions that the assessors can support with evidence.

3. How comparable are conditions across sessions and locations?

Two candidates can complete the same exercise and face substantially different demands.

Imagine a simulated performance conversation. One role player accepts the candidate’s suggestions readily. Another challenges every proposal. Or consider a case presentation where one panel uses helpful follow-up questions to guide a candidate towards an overlooked issue, while another challenges the assumptions behind each answer.

These differences affect the opportunities candidates have to demonstrate the relevant behaviour. Oliver and colleagues (2016) found that the interpersonal disposition portrayed by role players influenced participants’ behaviour, even when the underlying task remained the same. Their findings also highlight the value of piloting whether role players deliver the intended behaviour consistently.

The warning sign is uncontrolled variation in how an exercise is delivered. Shared materials are a starting point. Role players also need guidance on how to respond, and assessors need clear rules for questions, prompts and assistance.

One practical approach is to build planned prompts, or “cues”, into the exercise. In a role play, a question such as “How would you put that into practice with your team?” can create an opportunity to demonstrate planning. The exercise guide should specify when and how such prompts are used, and assessors should understand the behaviour they are intended to elicit (Kleinmann and Ingold, 2019).

Consistency should extend across locations, assessor teams and successive rounds. When local adaptations are needed, their effect on the demands of the exercise should be considered.

This still leaves room for responsive interaction. A good role play should respond to what the candidate does. The task is to define which responses are appropriate within the exercise and which would change what is being assessed.

4. Does the same rating mean the same thing to every assessor?

Assessor training often covers the exercises, the competency definitions and common rating errors. The crucial next step is practising how to apply the performance standards.

Suppose a rating of “3” means “meets the requirements” to one assessor and “average compared with today’s candidates” to another. Even a pair of assessors who agree after discussing an exercise may interpret the scale differently from the pair assessing the next candidate.

Frame-of-reference training develops a shared understanding of performance. Assessors rate concrete examples, compare their judgements with a justified reference standard and work through the reasons for differences.

Schleicher and colleagues (2002) compared two equally long training programmes that both included materials and practice ratings. The programme that also included discussion and feedback improved several aspects of rating quality. More recently, Gorman and colleagues (2024), using videos from an executive development assessment centre, found that FOR training helped assessors identify lower performance and reduced the influence of individual rating tendencies.

The warning sign is that assessors have received the scales but have never demonstrated a shared understanding of them. Ask what examples they have rated, what differences emerged and how those differences were resolved. The relevant standard needs to be shared across the assessor pool, including people who do not routinely work together.

5. What information do assessors receive before observing a candidate?

A candidate arrives at the assessment centre with a personality profile, cognitive test results and evaluations from earlier interview stages. Should every assessor see all of it before the first exercise?

Consider an assessor who has already read that a candidate lacks assertiveness. When watching a role play, they now have an interpretation available before collecting their own evidence. A positive recommendation from an earlier interviewer could provide a different starting expectation.

The potential influence of advance information has long been discussed in assessment centre practice. Kanning and colleagues (2007) raised this concern in their survey of German organisations. It is a reason to design the flow of information deliberately, rather than assume that more information is always helpful at every stage.

The warning sign is advance information being shared without a clear purpose for the assessor’s task. A CV can be useful for a biographical interview. Test results may be needed for an interview specifically designed to explore them. Neither automatically belongs in the briefing for someone whose job is to observe an independent exercise. The same question applies to photographs and personal details: what does this assessor need them for?

My recommendation is to keep exercise ratings independent of prior test results and interview evaluations wherever possible. Bring those sources together once the observations have been recorded and the exercise ratings have been completed.

This makes the contribution of each source easier to examine. If several assessments point in the same direction, you want to know that they provide separate evidence, rather than several versions of the same initial impression.

6. Does your assessment centre have a hidden agenda?

Imagine a candidate halfway through preparing a case presentation when the organiser announces that the preparation time has been cut short. The change is deliberate: the assessors want to see how the candidate handles pressure. It was not part of the task as explained, but the response will now influence the evaluation.

Before adding a test like this, the organisation should be able to explain which requirement of the role it assesses and why this particular situation provides useful evidence.

If handling sudden changes and maintaining effective performance under pressure are important job requirements, a carefully designed simulation may be appropriate. That includes infrequent but critical situations. Simply adding pressure, however, does not establish that the resulting behaviour measures resilience or any other intended competency.

Research on assessment centre design emphasises the interaction between the situation and the behaviour it elicits (Kleinmann and Ingold, 2019). That makes the choice of challenge part of the measurement itself.

The warning sign is an additional test justified mainly by “we wanted to see how they really behave”. The challenge should have a clear connection to the requirements analysis, defined observation opportunities and a usable scoring framework.

Participants need an honest understanding of the assessment process, although this does not require revealing every development within a simulation. SIOP (2018) highlights clear and consistent information about administration and time parameters. Covertly turning breaks or unrelated encounters into assessment activities creates a different problem: behaviour is being judged outside the agreed assessment framework (Kanning et al., 2007).

A demanding exercise can be useful. Its demands need to earn their place in the design.

7. What evidence supports your confidence in the results?

“We have used it for years.” “Our managers trust it.” “Candidates give us excellent feedback.”

All three may be true. None, on its own, establishes how well the assessment supports the decisions made from it.

Candidate feedback is valuable for understanding the experience of the process. Research also shows that reactions depend on when they are measured and on characteristics candidates bring to the situation (McCarthy et al., 2017).

I would also be cautious about treating positive candidate feedback as an unfiltered verdict. While a selection decision is still pending, candidates may have little incentive to criticise the people assessing them.

The warning sign is confidence supported mainly by acceptance and familiarity. A useful evaluation asks several questions: Do the tasks fit the requirements? Do the exercises and rating scales work as intended? How dependable are the ratings? What evidence connects the conclusions to subsequent performance?

For many organisations, a large-scale predictive validation study will be difficult to conduct. There are other ways to build evidence, including an examination of the assessment content, structured pilots and a careful assessment of whether existing research applies to the intended use (SIOP, 2018).

That last step matters. Sackett and colleagues (2022) emphasise that an employer cannot simply assume the average validity reported in research applies to its own setting. Evidence needs to support the particular claims being made about the procedure.

Can you explain the evidence behind your assessment centre?

The seven questions point to a common issue: whether the connection between the role, the exercises, the observations and the final judgement is clear and defensible.

An established procedure may have convincing answers to some questions and gaps in others. Identifying those gaps makes it possible to decide what deserves attention, which strengths should be retained and where a change would add value.

If you would like an independent view of your existing assessment centre, I would be happy to discuss it with you. Through my AC Audit, I review assessment materials and documented processes against established research findings, helping you understand how well the design is supported and where improvements would be worthwhile.

References

Gorman, C. A., Jackson, D. J. R., Meriac, J. P., Himmler, J. R., & Contreras, T. F. (2024). Beyond rating accuracy: Unpacking frame-of-reference assessor training effectiveness. Industrial and Organizational Psychology, 17, 206–219. Article

Kanning, U. P., Pöttker, J., & Gelléri, P. (2007). Assessment Center-Praxis in deutschen Großunternehmen: Ein Vergleich zwischen wissenschaftlichem Anspruch und Realität. Zeitschrift für Arbeits- und Organisationspsychologie, 51, 155–167. Article

Kleinmann, M., & Ingold, P. V. (2019). Toward a better understanding of assessment centers: A conceptual review. Annual Review of Organizational Psychology and Organizational Behavior, 6, 349–372. Article

Lievens, F., & Sackett, P. R. (2017). The effects of predictor method factors on selection outcomes: A modular approach to personnel selection procedures. Journal of Applied Psychology, 102(1), 43–66. Article

McCarthy, J. M., Bauer, T. N., Truxillo, D. M., Anderson, N. R., Costa, A. C., & Ahmed, S. M. (2017). Applicant perspectives during selection: A review addressing “So what?,” “What’s new?,” and “Where to next?” Journal of Management. Article

Oliver, T., Hausdorf, P., Lievens, F., & Conlon, P. (2016). Interpersonal dynamics in assessment center exercises: Effects of role player portrayed disposition. Journal of Management, 42(7), 1992–2017. Article

Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068.

Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87(4), 735–746. Article

Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures (5th ed.).