Examining the Effect of Interviewers on Longitudinal Survey Response Rates and Approaches to Improve the Hiring and Retention of High-Quality Interviewers

Importance of Measuring and Accounting for Interviewer Effects

Interviewers can vary in demographic attributes (e.g., age, gender), level of perceived experience and professionalism, approach to recruitment and survey administration (e.g., adherence to verbatim question wording), and behavior (e.g., reactions to reluctant respondents). Variability among interviewers can affect response rates, answers to survey questions, and data quality. In addition, interviewer effects can vary among groups of respondents (e.g., larger effects among older respondents compared to younger respondents). Thus, measuring, modeling, and accounting for interviewer effects is a crucial component of large-scale survey-driven studies.

Rising interviewer turnover and difficulties recruiting new interviewers are significantly increasing study costs and threatening the face-to-face interview paradigm used by many large-scale health and aging studies. Multiple organizations and federal agencies have responded with new strategies to improve hiring and retention of interviewers, including shortened hiring processes (i.e., reduced time between initial application and starting work), increased pay, bonuses and incentives, and restructuring of interviewer positions to regular, full-time positions with benefits. The effectiveness of these strategies has been mixed, in part because interviewers vary in their needs (e.g., some prefer full-time employment while others prefer short-term, flexible employment). Moreover, the impact of these strategies likely varies based on labor markets.

Changes to Interviewer Roles in Large-Scale Studies

The COVID-19 pandemic and increasing interviewer turnover have led many large-scale health and aging studies to change survey administration modes (e.g., video interviewing, multimodal administration). Many studies and surveys are also assessing new roles for field interviewers, including greater involvement of interviewers in survey design and data cleaning, as well as specialization among interviewers (e.g., video interviewing specialists).

Research Gaps and Opportunities from the Expert Meeting

• Increase monitoring of interviewers to better assess and analyze effects of interviewer behaviors on study response rates and survey responses.
• Improve interviewer training to reduce the impact of interviewer effects on data quality.
• Strategies for improving hiring and retention of interviewers. Provide training to interviewers to handle increasing nonresponse rates and respondent reluctance (e.g., refusal aversion training).
• Restructure interviewer roles to have increased involvement in study design, including assessing usability of survey instruments and assistance with data cleaning.
• Conduct additional research on the tradeoffs associated with offering regular hours and benefits to reduce interview turnover.
• Use proxy cost indicators (e.g., number of contact attempts) in existing health and aging surveys to monitor interviewing costs and effectiveness in real time.
• Increase multimodal and video administration to reduce reliance on face-to-face interviews.
• Assess which respondent populations (e.g., older respondents) benefit the most from face-to-face interviews.
• Deploy select, small pools of field interviewers for specific respondent populations and respondent recruitment.

Future Research Directions

To further understand interviewer effects and to develop strategies for improving interviewer hiring and retention, researchers may conduct additional studies on:

• Interviewer effects, including how effects differ between survey administration modes (e.g., in-person, video interviews).
• Impacts of change in survey administration modes on survey representativeness, particularly inclusion of underrepresented groups.
• The effectiveness of training for addressing workplace challenges frequently encountered by field interviewers (e.g., increasing nonresponse rates, respondent hostility).
• Strategies to increase retention and job satisfaction of interviewers, particularly study designs that avoid potential survivorship bias.
• Approaches for improving respondent scheduling, data quality, and survey representativeness in video interviews.

If interested in requesting the full workshop report, please e-mail Shanna Breil ( shanna.breil@nih.gov ).

September 11, 2023

10:00 a.m. – 10:15 a.m. | Welcome; Overview of current issues from NIA perspective; plans regarding key NIA surveys

• Melissa Chiu, CNSTAT
• Brady West, Moderator, University of Michigan
• John Phillips, NIA-BSR

10:15 a.m. – 11:40 a.m. | Session 1: Effects of Interviewer Characteristics and Skills on Response Rates and Data Quality

Overview of what is currently known

• The Past, Present, and Future of Research on Interviewer Effects, Kristen Olson and Allyson Holbrook
• Lessons from the agency’s vast experience with large scale face-to-face interviewer workforce, Timothy Olson
• Research findings on approaches to examining variation in interviewer effects and other factors across subgroups of respondents; lessons from the European Social Survey, Geert Loosveldt

Technology considerations: e.g., how does moving from in-person to computer- assisted and other video interviewing techniques relate to data quality? How is rapport between interviewer and respondent effected? Can these technologies be usefully implemented during the study enrollment phase?

• Innovative design approaches to improving the respondent experience, respondent retention, and data quality, Fred Conrad and Andrew Hupp
• UK experience with live video interviewing and other methods, Martin Wood and Gerry Nicolaas

Next steps and takeaways

• Survey of Health Ageing and Retirement in Europe, David Richter
• Designing experiments to identify key factors in the interviewer profile, survey outcome relationship. Approaches to facing similar challenges regarding interviewers, compounded by the multiple-country problems, Michael Bergman
• Considerations for specific NIA surveys (HRS, PSID, and other high-priority surveys), Colm O’Muircheartaigh

Open Discussion

11:40 a.m. – 1:45 p.m. | Session 2: Achieving optimal work force quality for in-person data collection. Recruitment, hiring models, retention issues, the roles of training and job/career paths

Current best practices in survey operations: Designing interviewer jobs to attract/retain best performers and to improve study quality

• Between Gig Work and Guaranteed Hours: Opportunities, Challenges, and Interviewer Perspectives, Grant Benson
• Project directors’ perspectives. The relationship between training mode (all virtual, mostly face-to-face, or blended) and new interviewer attrition. Comparative attrition rates for new and experienced interviewers, Brad Edwards
• Experiences with several large surveys, including the National Survey of Family Growth (NSFG), which faced challenges with face-to-face data collection post COVID-19, Andy Peytchev
• Role of supervisors/management. Importance of having front-line supervisors focused on supporting interviewers, and to pay interviewer bonuses, Kyle Fennell and Juan Mendizabal
• Cost comparisons across organizations, and relationship with interviewer hours and other aspects of the job, James Wagner
• The Census example: agency’s recruiting, hiring practices, as well as efforts to improve retention of interviewers—e.g., the role of benefits; differences with private sector survey firms; lessons from move to ACS from decennial, from SIPP move to annual from quarterly interview schedule, John Baker

Open Discussion

1:45 p.m. – 2:15 p.m. | Wrap-Up: The Interviewer workforce of the future and the broader context of a deteriorating field survey paradigm

Challenges and future research needs—near- term solutions, longer-term projects, Jennifer Dykema

Open Discussion

2:15 p.m. Adjourn

DBSR