Generative research

Improving internal government hiring systems

This case study explores the staff user experience of a third-party vendor management system through discovery research.


Recruitment, research facilitator, data analysis, report writing, stakeholder communication



Senior Service Designer (oversight), Intern, Experience Design Co-op (me)


14 weeks

Project Overview

The problem space

This user research project revolves around an internal contracting service that helps procure talent across the Ontario Public Service (OPS). A new vendor management system was recently implemented to support all requests' life cycle and reporting. At the Ontario Digital Service, our role was to conduct an internal discovery research project to shed light on the current user experience and help inform future decisions on improving the service for the staff.

Research Limitations

There were some research restrictions due to circumstances outside of our control. We only had access to interviewing internally facing user groups. We could not access the vendor management system, so we could not see visually how it works.


Research limitations can seriously hinder a research project depending on how we view it. We crafted our research guide to ensure all possible avenues were covered. I leaned on my facilitation skills to uncover as many insights as possible. We also emphasized limitations in our final report and the opportunity to do further research in the future.

User research and recruitment plan

This user research project began in the winter of 2023. The team started the user research plan, but it had to be paused for reasons outside their control. In May of 2023, I joined and resumed the work where they had left off. My initial responsibilities included reviewing the project background, finalizing the user research plan, and preparing facilitation guides and screener surveys.

Continuing where the previous co-ops left off

Why 1:1 semi-structured interviews?

We would have preferred to conduct usability testing to better understand how users interact with the vendor management system. However, since we didn't have access, we opted for 1:1 semi-structured interviews. We also chose not to use focus groups to avoid conformity bias.

Recruitment and user interviews

The opportunity to handle lead responsibilities early on

This was the first project at the Ontario Digital Service (ODS), and I was fortunate to be given the opportunity to take the lead on the recruitment process as well as facilitate almost all of the interviews.

Obtaining user group contact information

We finalized the user research plan with our partners and began recruiting. During our weekly check-ins with the partners, we discussed our need for contact information, and they provided us with over 800 email addresses. My responsibilities included creating email templates, contacting user groups, and scheduling interview sessions.

Challenges in the surprising amount of rescheduling

Perhaps it was a busy time for this user group, resulting in 15+ reschedules or no-shows and the 22/800 screener response rate. To counter this challenge and ensure timely in-scope delivery, I set up timely follow-up emails while keeping an eye on the mailbox to quickly accommodate rescheduling.

The pursuit of achieving flow state in user interview moderation

Flow state is a mental state when someone is fully immersed in using a combination of their skills while being challenged in a task or activity. Understanding that I would be facilitating most of these interviews, I sought feedback from management and colleagues to improve my skills on the go.

Synthesis and analysis

Starting affinity mapping each user group

Reflecting on the interviews, we found that the user groups were closely related, so the intern and I set up several synchronous meeting sessions to ensure proper and consistent analysis alignment. We chose affinity mapping to uncover broad themes from a significant amount of data. After filling any gaps in the interview notes, The intern and I began affinity mapping the 5 user groups one user group at a time. This process helped us condense 22 interview frames into 5 affinity maps.

Comparing affinity maps of five user groups

To analyze the user data effectively, we ran a thorough comparison of the five user groups. Through this process, we identified common themes and insights across the groups, which we have compiled and presented in a separate section named global findings.

What happens when everything becomes a global finding? 

As time passed, there became more global themes than unique themes for user groups. I learned that this can sometimes happen when a service and its users' roles overlap, and that's okay. We took this understanding to inform out report writing approach.

Report writing considerations

reflecting on the core problem

For this user research project, our job was to shed light on the current user experience through insight and recommendations to inform strategic decisions moving forward. My responsibilities at this stage included compiling our insights into a cohesive report presentation for the lead to craft the recommendations.

Structuring the report presentation

Sometimes, structuring the report according to the service steps can be effective. Since our findings became almost all global themes, we structured our report based on those. Further, the partners were familiar with the process, so they could digest the information this way.

What might advice look like with access to one side of the coin?

Due to restrictions outside our control, we could not see the product in action. This restriction meant we could only paint the picture using participant feedback in our interviews. That being said, the recommendations were a combination of broad and specific. It was an excellent opportunity to learn how to craft recommendations with a variable lens.

Presenting to stakeholders

Different presentation styles can build stronger relations.

At the beginning of the presentation, we asked the partners if they would like a formal or more conversational presentation style. They picked the latter, and we worked together to figure out potential solutions and next steps, which I don't think we would have gotten if we presented more formally.

What I learned

The value of pausing in interviews

Early on, when starting out moderation, I always tried to find the next follow-up question so there would be no awkward silence. One time, I couldn't find a follow-up question after the participant finished their thought. They elaborated on their point when I thought things would get uncomfortable!

Anticipate hurdles but understand that we can’t predict everything

I've learned that something always comes up for recruitment, no matter how much we prepare. If possible, adding a week or two to the recruitment timeline is a safe bet.

Participant preparedness

This user group has had a little over a year with this new vendor management system, so they have a lot of thoughts built over time. Some participants even had notes prepared. In one interview, I asked maybe five questions in total. So long as things are relevant, it’s okay to go off-script!