LESSONS FROM OTHER FIELDS
Are there parallels between hiring and child protection processes that can help reduce bias and narrow child welfare’s front door?
By Aaron Sojourner, Associate Professor, Department of Work and Organizations, Carlson School of Management, University of Minnesota
The mission of child welfare agencies is to ensure children are safe with their families. A long practice history and academic literature in child welfare grapples with better and worse ways to do this. This brief examines parallels that exist between organizational strategies for staff hiring and for child welfare decision-making. Why look here? Organizational hiring processes face a structurally similar problem to frontline decision-making in child protection: determining the most productive moments to deploy limited resources and requiring an ensemble of strategies to identify, assess, and act with respect to mission. Although the processes are not the same — for instance, hiring processes operate with a high degree of voluntary consent while child protection processes more often involve involuntary, coercive elements — even imperfect analogies can spark new insights.
A parallel process
In both hiring and child protection, the process includes an initial stage with ways to screen a broad range of individuals or their alleged actions. The need for breadth — so as not to miss highest risk cases — necessitates that initial strategies be inexpensive on a per-case basis. The basic goal here is to recognize cases with a high probability of being high-value to advance to a next stage for deeper inquiry, while not advancing cases with a low chance. There will always be errors of both types: missing cases that should advance and advancing ones that shouldn’t. The costs of committing and of avoiding these two types of errors differ.
Later stages of the process focus investment on gaining a deeper understanding of a narrower set of cases with the goal of determining the most appropriate level and type of resources (time and money) to invest. At each stage of inquiry, the process must balance the cost of processing cases against the beneficial value of information elicited, judged in terms of improving its use of resources towards the overall mission. The table below briefly illustrates a parallel sequence of strategies in hiring and in child protection:
Screening: Broad & Shallow (inexpensive per case)
Hiring
- Recruitment: Identify, engage with, and drive applications from the most promising potential employees and discourage them from the least promising.
- Application: Baseline inquiry to elicit basic info: Should we interview or decline?
Child Protection
- Awareness: Train mandatory reporters, community workers, and public and staff hotlines to stimulate referrals regarding most impactful cases and reduce referrals based on bias or misunderstanding.
- Hotline reports: Initial screening: Should we screen the report in for an investigation or assessment?
Assessment: Narrow & Deep (expensive per case)
Hiring
- Interview: Generate best signals before hire: Should we hire or decline?
Child Protection
- Investigation: Careful, thorough discovery and consideration of family system: Should we offer services? Remove child? Refer for criminal complaint?
Confirmation: Targeted Investment (deploy ongoing resources)
Hiring
- Hire: Negotiate package of compensation and responsibilities.
Child Protection
- Case planning and management: Offer or require supports and services, child removal and foster care.
Screening
In hiring, recruiting is the first stage in the process and aims to generate applications from individuals predicted to be a good fit for the position, while not generating applications from those predicted not to be. Some organizations do barely any recruiting at all. They put a sign in the window or a notice online and wait passively for applications to arrive. Better organizations actively seek out and engage promising talent and encourage their applications. In child protection, hotline reports are analogous to applications: agencies passively wait for allegations of maltreatment, and the response is reactive.
Mandatory reporting laws are designed to generate referrals from people in the best positions to know; yet most reports to the child protection hotline are not ultimately substantiated, suggesting that the current approach is not effectively identifying those children most at risk of abuse or neglect. Strategic investments in marketing and outreach, training, and technological innovation can promote the effectiveness of these initial intake systems. Reports are signals that contain an unknown combination of truth and noise, including bias. By looking at which sources tend to produce better and worse signals historically, the organization can learn which sources to invest in and which need further improvement.
Questions for child protection leaders to consider
- Who has good information about the identity of your high-risk cases?
- How can they be encouraged to let you know?
- What modes of communication, motivation, and trust-building are working or not?
- What leads low-value cases to be referred to you?
- Are there ways to limit this without significantly compromising the identification of high-risk cases?
- What are your worst systematic blind spots and barriers? How can you guard against that?
- In your mandatory reporting network, which hospital’s referrals had the highest screen-in rates over the last two years and which had the lowest? (Same for schools, law enforcement agencies, etc.)
- Among children on Medicaid, for which state records of hospital visits exist, which hospitals had the highest rate of failures to report cases that later entered the child protection system through other routes, or should have?
- Are there racial and ethnic disparities in these error rates?
Assessment
Gathering the best information
The next few stages each elicit progressively richer information on a narrowing set of high-value cases where the organization will invest the most resources in pursuit of its mission. No investment or less intensive investments are judged appropriate for other cases that do not meet the threshold for being screened-in. In hiring, a job application process is designed to solicit information from applicants in order to decide on whom to expend resources to gather more information through an in-person (or video) interview. Designing an application process that elicits more information makes it more costly (time and resources) for individuals to apply and for the organization to process, but can improve its ability to discern an applicant’s promise. Similarly, the information from interviews is used to decide whom to hire.
The best organizations deliberate up front and build a shared understanding about what kinds of factors matter and why, develop strategies for differentiating candidates on these dimensions, and know how to interpret evidence that will arise. They design structured interview processes with a pre-specified set of questions and create realistic job previews meant to generate information from a setting as close as possible to the relevant context for the work. They anticipate and navigate applicants’ impression-management strategies. Longer structured interviews appear to reduce the effectiveness of impression-management strategies.1
In child protection, the assessment or investigation phase is not typically viewed or experienced as a positive or protective experience, yet it is critical for determining whether the allegations of maltreatment are in fact true. Many child protection agencies utilize safety or risk assessments to fairly and comprehensively assess families in a consistent way to determine whether services should be offered, and whether the child needs to be removed.
Questions for child protection leaders to consider
- Do your staff share a common understanding of relevant factors, how to recognize them in practice, and how to translate evidence into decisions?
- Does your organization use consistent, valid assessment tools?
Reducing bias in decision-making
At each stage, best practices in hiring will maximize consideration of relevant evidence and minimize consideration of irrelevant factors. When screening individuals to become U.S. Air Force pilots, tests focused on generating relevant evidence about how individuals would react during “critical incidents.” For instance, the tests simulated physical disorientation on the ground before putting people in control of multi-million-dollar jets.2 This principle of maximizing consideration of relevant evidence has led many organizations to design realistic job previews and situational judgment tasks — under conditions that mimic as closely as possible those they will face on the job — to surface relevant evidence.3
The best hiring practices also strive to minimize consideration of irrelevant factors, including biases. For instance, when an orchestra began having musicians audition behind a screen, it started hiring more women.4 The screen allowed the organization to maximize consideration of relevant evidence — the quality of the musical performance — and minimize consideration of irrelevant factors, such as candidate appearance or gender. They designed a process that better elicited and centered what mattered and better suppressed what did not.
Similar examples exist within the child protection system. The overrepresentation of Black children at every decision point within the system is well documented. These disparities are often attributed to longstanding institutionalized discrimination and biases toward Black families. To remove the potential for bias in decisions about whether to remove children from their homes, Nassau County, N.Y.’s Health and Human Services Agency implemented a blind removal process, whereby staff de-identify the case file and present details without any mention of demographic information that may elicit bias. Focusing on the information most relevant to the case resulted in fewer Black children removed from their homes.
Questions for child protection leaders to consider
- If responsible staff were presented with the same information about a case, in what share of cases would they reach the same judgment about how to handle it?
- Would the basis of these decisions be similar?
- To what extent are decisions determined by unintentional, idiosyncratic factors — or biases — rather than intentional, shared factors?
Shared decision-making
Each person involved in the hiring or assessment process has individual idiosyncratic biases and can only get an imperfect signal of a candidate’s value to the organization. Having a panel of multiple people independently consider the information about a candidate can be expensive because it requires more staff time.
However, aggregating over multiple people’s independent judgments will tend to get closer to the truth if their individual biases tend to cancel each other out. Having each staff member review evidence, and develop and submit preliminary assessments independently, guards against groupthink based off others’ analytic effort and increases the chances of gaining benefit from multiple evaluations. It can also promote more fruitful, shared deliberation afterward. When everyone develops assessment simultaneously, attention wanders and herding can occur. There are costs, though. Interviewees do not want to sit through the same kind of interview many times. Video recording may help in this context, as it allows for multiple, independent, asynchronous assessments. Similarly, child protection staff assess, investigate, and adjudicate each situation, and group decision-making promotes more consistent judgment, and more effective outcomes. Further, investing in the quality and capacity of child welfare supervisors is critical for supporting caseworkers and enhancing the quality of their assessments.
Questions for child protection leaders to consider
- Does it make sense to have additional individuals evaluate each situation before an official decision? Each independently before deciding together?
- Can the organization pre-specify rubrics of evaluation that each individual uses to assess a situation before confirming a joint decision?
- Can the organization summarize assessments across individuals in a routine way to promote quick recognition of areas of agreement and disagreement?
Confirmation
Ultimately, the purpose of these evaluation processes is to allocate the organization’s scarce resources across cases. In the hiring context, that means choosing whom to hire and with what rights and responsibilities. It can also mean using organizational resources to signal continuing interest in some candidates who have potential as future hire, even though they are not selected for the current position. For child protection agencies, it means choosing what services and supports to offer each family based on what they need to keep children safe and with their families whenever possible.
In each of these distinct contexts, the organization has a set of tools at its disposal. Each tool has costs and benefits that depend on the specific instance in which it is used and the organizations’ use of tools is limited given budget and operational constraints. Mistaking a screw for a nail and using a hammer instead of a screwdriver is a costly mistake. All the screening and assessment work in the prior steps feed into these downstream decisions to try to deploy the set of tools to their highest, best use. In hiring, organizations aim to ensure that the set of job duties and compensation fit together well to attract and retain the best set of individuals to work as a team and make the organization successful. Child protection agencies must strive to ensure that the mix of effective decision-making, provision of family supports and service array, and policies to ensure timely permanency are aligned to promote the well-being of children and families.
1. Levashina, J., Hartwell, C.J., Morgeson, F.P., & Campion, M.A. (2014). The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature. Personnel Psychology, 67, 241-293.
2. Flanagan, J. (1954). The Critical Incident Technique. Psychological Bulletin, 51(4), 327-358.
3. Whetzel, D. & McDonald, M. (2009). Situational Judgment Tasks: An overview of current research. Human Resource Management Review, 19, 188-202.
4. Rice, C. (October 14, 2013). How blind auditions help orchestras to eliminate gender bias. Retrieved from https://www.theguardian.com/women-in-leadership/2013/oct/14/blind-auditions-orchestras-gender-bias
Aaron Sojourner is a labor economist and associate professor at the Carlson School of Management, University of Minnesota. His research focuses on three areas: (1) effects of labor-market institutions on economic productivity and in politics, (2) policies to promote efficient and equitable development of human capital with a focus on early childhood and K-12 education systems, and (3) behavioral economic approaches to consumer financial decisions.