Automate psychometric tests norms computations and updates to improve efficiency, accuracy and relieve researchers
An Airflow environment with workflows performing ETL and computing psychometric tests norms
A 4-stage process performed by a multi-disciplinary team of Movify consultants and Cebir employees
Substantial gains in efficiency, reliability and internal satisfaction of the research team
While absolute results from psychometric tests provide valuable information about a candidate’s capabilities, relative results are at least as important. Comparing a candidate’s performance to a normative group with a similar profile (such as education level, native language, etc.) ensures a fairer and more contextually accurate evaluation. This approach accounts for potential biases and natural variations within different demographic groups, making it easier to identify truly exceptional candidates and avoid misjudging potential due to irrelevant differences. Relative comparisons enhance the fairness and precision of the recruitment process, leading to better hiring decisions.
Keeping up-to-date norms (i.e. statistics for each normative group) is essential to ensure fair and accurate interpretation of psychometric test results. As populations evolve, outdated norms can misrepresent a candidate’s performance and introduce bias. Regularly refreshed data ensures comparisons reflect current standards, allowing for more relevant, equitable, and reliable hiring decisions.
Cebir helps businesses make smarter, data-driven hiring decisions by offering psychometric assessments. Psychometric tests play a crucial role in recruitment by providing objective and standardised measures of a candidate’s cognitive abilities, personality traits, and behavioural tendencies. Unlike interviews or resumes, which can be influenced by bias or presentation skills, psychometric assessments offer data-driven insights that help employers predict job performance and cultural fit more accurately. By incorporating these tests into the hiring process, organisations can make more informed decisions, reduce turnover, and build stronger, more effective teams.
Cebir wanted to automate the update of norms, which used to be performed manually by researchers. It ensures norms are consistently refreshed with the latest test results, reducing the risk of outdated benchmarks and human error. Automation also saves time and resources, enabling faster integration of new data and more frequent updates. This leads to more accurate, dynamic, and scalable comparisons across candidate profiles, ultimately improving the reliability and fairness of psychometric assessments in recruitment. Also, by relieving researchers from repetitive manual tasks, it allows them to focus on more meaningful and intellectually engaging work, such as developing new assessment tools or refining test validity.
Following the introduction of our automation tool, researchers executing several scripts in sequence and copy-pasting intermediate results is now a thing of the past. They can now focus on more meaningful research tasks and use the developed solution to assist them in their day-to-day tasks.The solution is an Airflow environment with a set of data workflows which can be split into 3 main categories:
1. operational data extraction
2. operational data preprocessing
3. norms computation and update
Workflows of the first 2 categories form the hidden part of the iceberg, they are scheduled automatically and are not of direct interest for researchers, they’re here to facilitate computations down the line.
The tip of the iceberg contains the workflows of the last category. Those can be triggered on-demand by researchers, usually when norms need to be updated for a specific psychometric test. Upon successful completion of the workflow, updated norms and a detailed research report showcasing the different statistics are made available to the researcher who requested it. The researcher can then review and validate the new norms so that they can be applied in production.To a lesser extent, the solution can also assist in performing ad-hoc research tasks because it can provide data extracts the researchers can then import in their statistics software of choice to answer specific questions.
.png)
The team was composed of 1 software engineer with data science experience from Movify working in close collaboration with 1 product owner and 2 researchers from Cebir. The project was organised in 4 sequential stages, explained more in depth in the following sections.During the implementation stage, the team regularly sat together to show progress and gather feedback. Keeping a short feedback loop was important to brainstorm potential improvements, catch issues as early as possible and keep all stakeholders engaged by showing incremental progress over time.
Existing Processes Analysis
There existed scripts that were somewhat achieving the main goal. We didn’t want to reinvent the wheel, hence we took time to analyse them in order to identify what could be reused and what needed improving. The most important pain points we discovered were that they were written in proprietary technologies that were not easily automatable and that they required too much manual interaction to get to the end result.
Data Exploration
This stage focused on exploring the complex database to gain understanding on how the data was modelled and identifying the location of the required data so that it could be extracted and used properly.
Implementation
This was the main stage of the project, porting the existing sub-processes scripts into high-quality code that can be improved and maintained over time. The existing scripts were complex and careful thought has been brought to split them into more atomic parts focusing on one specific task. Then, we designed Airflow workflows orchestrating all these tasks, achieving the end goal without requiring manual interaction anymore.
Deployment
During this final stage, we brought the developed solution into the hands of the researchers. We also performed optimisations to handle bigger data volumes and set up some monitoring to alert the technical team when errors occur.
After 6 months of development, the research team started incorporating the developed solution into their everyday life. Cebir was starting to reap the fruits of their investment very soon after. Zooming in on a specific psychometric test, history has shown that updating this test took on average 3 weeks. With the new solution in place, this time was reduced to 1 day. This is a substantial efficiency gain and life improvement for researchers. Their involvement is now limited to reviewing and validating the produced results, instead of being responsible for the whole execution. Overall, the research team has been able to update psychometric tests faster than ever. In the first month the solution was in production they were able to update 30 tests, which would have taken around 10 to 14 months with the old way of working, showing real promise of the solution.
