Making a Difference An Impact Study of Big Brothers Big Sisters Joseph P. Tierney Jean Baldwin Grossman with Nancy L. Resch A Publication of Public/Private Ventures Making a Difference An Impact Study of Big Brothers Big Sisters Joseph P. Tierney Jean Baldwin Grossman with Nancy L. Resch A Publication of Public/Private Ventures Making a Difference: An Impact Study of Big Brothers Big Sisters Public/Private Ventures is a national nonprofit organization whose mission is to improve the effectiveness of social policies, programs and community initiatives, especially as they affect youth and young adults. In carrying out this mission, P/PV works with philanthropies, the public and business sectors, and nonprofit organizations. We do our work in four basic ways: • We develop or identify social policies, strategies and practices that promote individual economic success and citizenship, and stronger families and communities. • We assess the effectiveness of these promising approaches and distill their critical elements and benchmarks, using rigorous field study and research methods. • We mine evaluation results and implementation experiences for their policy and practice implications, and communicate the findings to public and private decision-makers, and to community leaders. • We create and field test the building blocks—model policies, financing approaches, curricula and training materials, communication strategies and learning processes— that are necessary to implement effective approaches more broadly. We then work with leaders of the various sectors to implement these expansion tools, and to improve their usefulness. P/PV’s staff is composed of policy leaders in various fields; evaluators and researchers in disciplines ranging from economics to ethnography; and experienced practitioners from the nonprofit, public, business and philanthropic sectors. Board of Directors Siobhan Nicolau, Chair President Hispanic Policy Development Project Amalia V. Betanzos President Wildcat Service Corporation Yvonne Chan Principal Vaughn Learning Center John J. DiIulio, Jr. Fox Leadership Professor of Politics, Religion and Civil Society University of Pennsylvania Susan Fuhrman Dean, Graduate School of Education University of Pennsylvania Matthew McGuire Director of Private Sector Initiatives Wildcat Service Corporation Michael P. Morley Senior Vice President Eastman Kodak Company Jeremy Nowak Chief Executive Officer The Reinvestment Fund Marion Pines Senior Fellow Institute for Policy Studies Johns Hopkins University Isabel Carter Stewart National Executive Director Girls Incorporated Mitchell Sviridoff Community Development Consultant Marta Tienda Professor of Sociology Princeton University Gary Walker President Public/Private Ventures William Julius Wilson Lewis P. and Linda L. Geyser University Professor Harvard University Making a Difference: An Impact Study of Big Brothers Big Sisters Foreword Over the past decade mentoring has gained enormous respect and support. In one respect that is not surprising: there is nothing so heartwarming, comprehensible and reassuring as an adult befriending and supporting a younger person. Mentoring also produces important results. In an era when large numbers of Americans have little confidence in social interventions, that mentoring produces hard outcomes for adolescents regarding drug use, violent behavior, school performance and family relationships is at least equal in importance to its intuitive appeal. And mentoring is undiluted social intervention: connecting two strangers of different age groups, supporting and monitoring their relationships through the medium of an organization created for and dedicated to making those relationships work—in the case of this study, Big Brothers Big Sisters of America. We are re-issuing this 1995 impact study of Big Brothers Big Sisters, in part as a reminder that young lives, even those with serious obstacles, can be profoundly affected by social intervention. The fact that many social interventions for young people have not produced strong results is not a sound basis for giving up on either interventions or adolescents—but is rather a reminder that affecting young lives in an enduring and positive way is very hard work. Like searching for oil or investing in startup companies, there are more failures than successes. Mentoring is like finding a gusher or having invested in America Online at the beginning; we should applaud its success, and use it for all its worth. For mentoring is both a discrete program, and a broader idea: that individual change and progress is fundamentally about having other individuals care, support, tend to and guide on a one-to-one basis. There is no substitute. The second reason for this re-issue is to remind all of us that this study did not show that mentoring, as a generic idea, is effective. This mentoring was carried out by Big Brothers Big Sisters: a sole purpose federation with almost a century of experience and a distilled-from-experience set of operational guidelines about screening, matching, training, supervising and monitoring. This experience results in mentoring relationships that are intense (weekly, multi-hour meetings) and enduring (over a year in length)—and effective. Mentoring, either as a discrete program or as an idea to inject in schools, afterschool programming or juvenile justice institutions, is neither cost-free nor a knock at professionals. Its easy attractiveness belies the effort and structure that makes it work. Neither warm-hearted volunteers nor well-intended professionals in schools can make it uniformly effective without tending to the lessons that Big Brothers Big Sisters has learned. Thanks very much to the national BBBSA organization and its current president, Judy Vredenburgh, to the local chapters that agreed to participate in the study, and especially to Tom McKenna, who was president of BBBSA when this study took place. Few leaders of established organizations are voluntarily willing to take the risk of an impact study; his willingness has provided useful information and guidance, and most of all, confidence that our youth with the most obstacles can be helped—now. Gary Walker President Public/Private Ventures September 2000 Making a Difference: An Impact Study of Big Brothers Big Sisters Making a Difference: An Impact Study of Big Brothers Big Sisters Acknowledgments This study was made possible by funding from Lilly Endowment, Inc., The Commonwealth Fund, The Pew Charitable Trusts and an anonymous donor. All the members of P/PV’s adult/youth relationships research team contributed to this report: Cynthia L. Sipe led the evaluation design and instrument development phases of the project, thus giving it a firm foundation; Nancy L. Resch expertly analyzed the data and drafted the appendices; and Kristine Morrow, Melanie Styles, Alvia Branch, Kathryn Furano, Phoebe Roaf, Danista Hunte and Chris Welser contributed knowledge developed through their work on our other three BBBS studies. Thomas J. Smith and Gary Walker helped to shape the executive summary. Michelle Alberti Gambone, Mark Hughes, Bernardine Watson, Marc Freedman, Jeffrey Greim, Natalie Jaffe and Carol Thomson thoughtfully reviewed the drafts of the report and contributed to its clarity. Sheena McConnell, Walt Corson and Allen Schirm of Mathematica Policy Research, Inc. assisted with the research design, and their colleagues Joy Gianolio, Cheryl DeSaw and Linda Gentzik managed the mechanics of the random assignment process and directed the interviewing effort. The project further benefitted from the wisdom of P/PV’s Adult/Youth Relationships Advisory Board and Research Advisory Group members. The following advisors provided direction in the study’s design, conduct and analyses: Anita Summers of the University of Pennsylvania; Henry Levin of Stanford University; Richard Danzig, an authority on youth service; Beatrix Hamburg of the William T. Grant Foundation; Harold Howe and Heather Weiss of Harvard University; Emmy Werner of the University of California; and Joan Schine of the Early Adolescent Helper program. The following advisors carefully reviewed drafts of the report and contributed to its clarity: Frank Furstenberg of the University of Pennsylvania; Robinson Hollister of Swarthmore College; Frank Levy of MIT; Marta Tienda of the University of Chicago; and Jacqueline Eccles of the University of Michigan. Alan Krueger of Princeton University also reviewed the report. We also wish to acknowledge the work of P/PV’s MIS, support and editorial staffs. Carol Dash expertly prepared the document and patiently worked with the authors; Batia Trietsch and Eleanor Hammond carefully processed the voluminous data; Angela Everman, Greg Weber and Donna Sulak ably handled the programming tasks; and Rhodie Bruce-Holly provided secretarial support. Joseph Zakrzewski offered continuing support and guidance. Michael Callaghan (copy editing), Maxine Sherman (word processing) and Carol Eresian (proofreading) produced the document. Of course, conduct of the study would not have been possible without the assistance and cooperation we received from Thomas M. McKenna, BBBSA executive director, and Dagmar McGill, BBBSA deputy executive director. We greatly appreciate their participation, and that of the executive directors of the eight agencies who agreed in 1991 to allow us to conduct research at their agencies: David Schirner (Columbus), Frank Ringo (Houston) Linda Anderson, (Minneapolis), Tom Weber (Philadelphia), Linda Searfoss (Phoenix), Elizabeth Callaghan (Rochester), Sharon Baughman (San Antonio) and Nick Mork (Wichita). Special thanks go to the research liaisons at the study agencies who served as the point-of-contact with P/PV: John Hamilton (Columbus), Peggy Turner and Fairan Jones (Houston), Michael Charland (Minneapolis), Cheryl Thomas (Philadelphia), Madeleine Stilwell (Phoenix), John Walker and Lori Vanauken (Rochester), Kathy Blizzard (San Antonio) and Janet Rhodes (Wichita). Most of all, we would like to thank the case managers who explained the research to the study participants and obtained their consent to participate in this research project. The case managers are: Columbus—Jill Clinger, Jill Gates, Nancy Johnson, Eve Koby, Kelli Mcauley, Michelle Mosher, Cheryl Perkins, Jean Rickly, Amy Rohling, Leta Slavik, Iben Smith and Kerry Welty; Houston—Angela Carruba, Clara Cooper, Jody Hopkins, Angela Koeppeh, Lisa Vaughan and Venetia Wilks; Minneapolis—Jennifer Conlon, Jenny Corniea and Carla Grayes; Philadelphia—Joel Cohen, Dionne Cosby, Lori Deluca, Sylvia Fields, Rebecca Gaspar, Christine Linvill, Erin McConaghy, Cheryl Potter, Douglas Powell, Dawn Siman, Stephen Smith, Terri Tinnin, Debbi Toy and Patricia Wells; Phoenix— Bernadette Alvarado, Teresa Bacon, Sandra Burke, Kevin Davis, Adrian Decker, Frank Delamater, Marcia Duggar, Lenora Forbes, Maryanne Frost, Mark Kimball, Caroline Marquez, Tracy Sallen, Deborah Smith, Tracy Sullivan, Pat Thomas, Susan Wiltfong and Lori Zimmerman; Rochester—Heron Allen, Doris Barr, Marshall Boyler, Mike Connelly and Gina Hurley; Wichita—Kendra Coop, Sue Friend, Diane Hirschfeld, Shari Hocutt and Jennifer Matson; San Antonio—Justine Flores and Kathy Jones. Finally, we would like to thank both the youth who participated in this project and their parents. Making a Difference: An Impact Study of Big Brothers Big Sisters Contents Executive Summary ii I. Introduction 2 The Nature of the Problem . . . . . . . . . . . . . . . . . . . . . . . . .2 P/PV’s Mentoring Research . . . . . . . . . . . . . . . . . . . . . . . .2 Organization of the Report . . . . . . . . . . . . . . . . . . . . . . . . .3 II. The Big Brothers Big Sisters Program 4 Operating Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 BBBS and the Mentoring Field . . . . . . . . . . . . . . . . . . . . . .5 Study Agency Selection and Description . . . . . . . . . . . . . . .7 III. Research Design 8 Hypothesized Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Design Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Matching Treatment Youth . . . . . . . . . . . . . . . . . . . . . . . .10 Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 IV. The Sample Youth and the Volunteers 12 Background Characteristics of Study Sample Youth The Treatment Youth . . . . . . . . . . . . . . . . . . . . . . The Volunteers . . . . . . . . . . . . . . . . . . . . . . . . . . . Length of Matches . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 . . . . . .16 . . . . . .16 . . . . . .16 . . . . . .19 V. The Impact on Youth of Having a Big Brother or Big Sister 20 Antisocial Behaviors . . . . . . . . . . . . . . . . . . . . . Academic Attitudes, Behavior and Performance Family Relationships . . . . . . . . . . . . . . . . . . . . . Peer Relationships . . . . . . . . . . . . . . . . . . . . . . Self-Concept . . . . . . . . . . . . . . . . . . . . . . . . . . Social and Cultural Enrichment . . . . . . . . . . . . . Summary of BBBS Effects on Youth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 . .23 . .25 . .26 . .27 . .28 . .29 VI. Summary and Conclusions 30 What Produced These Results? Can More Youth Be Served? . . Areas for Future Research . . . . Final Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 .31 .32 .33 Endnotes 34 References 36 Making a Difference: An Impact Study of Big Brothers Big Sisters Appendix A: Study Methods 40 Appendix B: Additional Tables 50 Tables 1. Characteristics of Study Agencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 2. Sample Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 3. Race/Gender and Age of Youth by Treatment Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 4. Characteristics of the Study Youth’s Households and Parents/Guardians . . . . . . . . . . . . . . . . . . . . . . . . .13 5. Stressful Life Experiences of the Youth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 6. Characteristics of Never-Matched Treatment Youth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 7. Demographic Characteristics of Volunteers by Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 8. Characteristics of the Matches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 9. Net Impact of Participation in BBBS on Initiating Use of Drugs and Alcohol . . . . . . . . . . . . . . . . . . . . . . . .20 10. Net Impact of Participation in BBBS on Hitting, Stealing and Damaging Property . . . . . . . . . . . . . . . . . . .21 11. Net Impact of Participation in BBBS on Academic Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 12. Net Impact of Participation in BBBS on Family Relationships Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . .24 13. Net Impact of Participation in BBBS on Peer Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26 14. Net Impact of Participation in BBBS on Self-Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27 15. Net Impact of Participation in BBBS on Social and Cultural Enrichment Outcomes . . . . . . . . . . . . . . . . . .28 16. How Youth Benefit from Big Brothers Big Sisters Relative to Similar Non-Program Youth 18 Months After Applying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 Appendix Tables A.1. Outcome Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 A.2. Internal Consistency of Scales Used as Outcome Measures, Assessed at Baseline and Follow-Up . . . . . .43 A.3. Descriptive Statistics of Scales Used as Outcome Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44 A.4. Explanatory Variables Used in Regression Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 A.5. Selected Baseline Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47 B.1. Net Impact of Participation in BBBS on Antisocial Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50 B.2. Net Impact of Participation in BBBS on Academic Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 B.3. Net Impact of Participation in BBBS on Relationship Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 B.4. Net Impact of Participation in BBBS on Social and Cultural Enrichment Outcomes . . . . . . . . . . . . . . . . . .54 B.5. Volunteer Screening Procedures by Agency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56 B.6. Match-Related Information by Agency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Appendix Endnotes 58 ii Making a Difference: An Impact Study of Big Brothers Big Sisters Executive Summary The past decade has seen widespread enthusiasm for mentoring as a way to address the needs and problems of youth—but no firm evidence that mentoring programs produce results. We now have that evidence. In this report, Public/Private Ventures (P/PV) provides scientifically reliable evidence that mentoring programs can positively affect young people. This evidence derives from research conducted at local affiliates of Big Brothers Big Sisters of America (BBBSA), the oldest, best-known and, arguably, the most sophisticated mentoring program in the United States. Big Brothers Big Sisters (BBBS) programs currently maintain 75,000 active matches between a volunteer adult and a youngster. Both the programs and matches are governed by carefully established procedures and criteria. P/PV conducted a comparative study of 959 10- to 16-year-olds who applied to BBBS programs in 1992 and 1993. Half of these youth were randomly assigned to a treatment group, for which BBBS matches were made or attempted; the other half were assigned to BBBS waiting lists. We compared the two groups after 18 months and found that participants in a BBBS program: • Were less likely to start using drugs and alcohol; • Were less likely to hit someone; • Improved school attendance and performance, and attitudes toward completing schoolwork; and • Improved peer and family relationships. This report is part of P/PV’s eight-year investigation of a range of adult-youth relationship projects. In other reports, we have examined program practices; volunteer recruitment and screening in BBBS programs; and the characteristics of adult-youth relationships in BBBS and other mentoring programs. An Effective Approach to One-to-One Mentoring The findings presented in this report reflect the workings of a carefully structured approach to mentoring. Understanding how BBBS programs operate and the standards they adhere to is important, since many other mentoring programs are not as well-structured or carefully managed as the BBBS programs whose matches we studied. Local BBBS programs are autonomously funded affiliates of BBBSA. In addition to providing ongoing support and representation for its affiliates, the BBBSA national office serves the critical function of promulgating criteria and standards that largely determine the development, maintenance and quality of local matches. To be formally designated a Big Brothers or Big Sisters program, local agencies must adopt these standards, with minor variations allowed to accommodate local characteristics. The standards govern the screening and acceptance of both youth and adults; the training and orientation volunteers must undergo; the matching process; required meeting frequency; and the ongoing supervision of matches, which involves regular contact between the agency and the adult volunteer, the youth and the parent. Most local programs operate in more or less the same way: they recruit and carefully screen volunteer applicants for one-to-one matches; they screen youth, who usually come from singleparent households and who must (along with their parents) desire to enter into a match; and they carefully match adult volunteers with youngsters based on backgrounds, on the stated preferences of adult volunteers, parents and youth, and on geographic proximity. On average, the adult-youth pair meets for three to four hours three times per month for at least a year. In cooperation with the national BBBSA office, P/PV chose eight local, accredited BBBS agencies for this study. We used two criteria in selecting agencies. The first was a large caseload; our aim was to select from the largest BBBS agencies so as to generate adequate numbers of youth for the research sample and to minimize the impact of research activities on agency operations. The second was geographic diversity. The selected sites represent most regions of the United States; they are located in Philadelphia; Rochester, New York; Minneapolis; Columbus, Ohio; Wichita, Kansas; Houston; San Antonio; and Phoenix. Executive Summary Study Design and Sample Youth Major Findings The sample youth were between 10 and 16 years old (with 93% between 10 and 14) when they were found eligible for the BBBS program. Just over 60 percent were boys, and more than half were minority group members (of those, about 70 percent were African American). Almost all lived with one parent (the mother, in most cases), the rest with a guardian or relatives. Many were from low-income households, and a significant number came from households with a prior history of either family violence or substance abuse. The overall findings are positive. The following are the most noteworthy results: Our research strategy was to compare youth who participated in BBBS programs with those who did not. Thus, we conducted baseline interviews with all youth at the time they were found eligible for the program, then randomly assigned them either to the treatment group, who were immediately eligible to be matched with adult volunteers, or to the control group, who remained on a waiting list for 18 months—a not uncommon waiting period among BBBS applicants. Both groups were re-interviewed 18 months later. Of the 1,138 youth originally randomized, 959 (84.3%) completed both baseline and follow-up interviews, thus becoming the sample on which findings are based. Of the 487 youth in the treatment group, 378 were matched with a Big Brother or Big Sister, and received the agency support and supervision that would typically be provided. The matched Little Brothers and Little Sisters met with their Big Brother or Big Sister for an average of almost 12 months, with meetings about three times per month lasting about four hours each time. The aim of the research was to determine whether a one-to-one mentoring experience made a tangible difference in the lives of these young people. We chose six broad areas in which we hypothesized that the mentoring experience might have effects, identified in large part through discussions with local program staff, and a review of the guidelines and other materials produced by the national BBBSA office. The six areas were antisocial activities; academic performance, attitudes and behaviors; relationships with family; relationships with friends; self-concept; and social and cultural enrichment. All findings reported here are based on self-reported data, obtained from baseline and follow-up interviews or from forms completed by agency staff. Analysis of these data involved multivariate techniques that compared the follow-up survey results for treatment and control youth, controlling for baseline characteristics.1 iii • Little Brothers and Little Sisters were 46 percent less likely than controls to initiate drug use during the study period. Our results indicate that for every 100 youth in this age group who start to use drugs, only 54 similar youth who have a Big Brother or Big Sister will start using drugs. An even stronger effect was found for minority Little Brothers and Little Sisters, who were 70 percent less likely to initiate drug use than other similar minority youth.2 • Little Brothers and Little Sisters were 27 percent less likely than controls to initiate alcohol use during the study period, and minority Little Sisters were only about one-half as likely to initiate alcohol use. • Little Brothers and Little Sisters were almost one-third less likely than controls to hit someone. • Little Brothers and Little Sisters skipped half as many days of school as did control youth, felt more competent about doing schoolwork, skipped fewer classes and showed modest gains in their grade point averages. These gains were strongest among Little Sisters, particularly minority Little Sisters. • The quality of relationships with parents was better for Little Brothers and Little Sisters than for controls at the end of the study period, due primarily to a higher level of trust in the parent. This effect was strongest for white Little Brothers. • Likewise, there were improvements in Little Brothers’ and Little Sisters’ relationships with their peers relative to their control counterparts, an effect most strongly evidenced among minority Little Brothers. We did not find statistically significant improvements in selfconcept, nor in the number of social and cultural activities in which Little Brothers and Little Sisters participated. iv Making a Difference: An Impact Study of Big Brothers Big Sisters Conclusions Our research presents clear and encouraging evidence that caring relationships between adults and youth can be created and supported by programs, and can yield a wide range of tangible benefits. The most notable results are the deterrent effect on initiation of drug and alcohol use, and the overall positive effects on academic performance that the mentoring experience produced. Improvement in grade point average among Little Brothers and Little Sisters, while small in percentage terms, is still very encouraging, since non-academic interventions are rarely capable of producing effects in grade performance. These findings, however, do not mean that the benefits of mentoring occur automatically. The research, as noted previously, describes the effects of mentoring in experienced, specialized local programs that adhere to well-developed quality standards. In our judgment, the standards and supports BBBS programs employ are critical in making the relationships work, and thus in generating the strong impacts we have reported. If such standards and supports can be duplicated, the expansion and replication of mentoring initiatives for early adolescents would appear to be a strong and sensible investment, from which at least several million youth could benefit. Yet this raises two critical issues. First, is there a sufficient number of volunteers who would be willing to make the time and emotional commitment? The indications from prior research are inconclusive. The second issue is that the support and supervision necessary for mentoring initiatives to produce effective matches cost money—roughly $1,000 per match. It is extremely unlikely that significant expansion could be accomplished entirely with private funds. Public funding also seems unlikely at this time, when budgets for social programs are being drastically cut at the federal level and social policy interventions are widely viewed by the public as ineffective. However, evidence of effectiveness like that contained in this report—especially around issues of drugs, violence and schooling—may influence the public’s view of what can be accomplished, and may also stimulate policymakers to begin shaping a new and more effective social policy approach for youth—one that focuses less on specific problems after they occur, and more on meeting youth’s most basic developmental needs. Making a Difference: An Impact Study of Big Brothers Big Sisters 1 2 Making a Difference: An Impact Study of Big Brothers Big Sisters Introduction For more than 90 years, the Big Brothers Big Sisters of America (BBBSA) network of agencies has created and supported oneto-one relationships between adult volunteers and youth living in single-parent households. Despite its long existence, however, the effects of this mentoring program on the lives of the youth have yet to be credibly documented. In this report, Public/Private Ventures (P/PV) provides the first scientifically credible evidence that Big Brother Big Sisters (BBBS) programs have many positive and socially important effects on the lives of its young participants. While this is good news to the mentoring field, the positive impacts presented in this report have implications that extend to youth policy in general. Participation in a BBBS program reduced illegal drug and alcohol use, began to improve academic performance, behavior and attitudes, and improved peer and family relationships. Yet the BBBS approach does not target those aspects of life, nor directly address them. It simply provides a caring, adult friend. Thus, the findings in this report speak to the effectiveness of an approach to youth policy that is very different from the problem-oriented approach that is prevalent in youth programming. This more developmental approach does not target specific problems, but rather interacts flexibly with youth in a supportive manner. The Nature of the Problem Support and guidance from adults are a critical part of the process that allows youth to grow into responsible adults. Yet today there is a scarcity of such support, especially among poor youth. The institutions we have historically relied on to provide youth with adult support and guidance—families, schools and neighborhoods—have changed in ways that have dramatically reduced their capacity to deliver such support. For example, there are fewer adults in families today: more than one in four children are born into a single-parent home, and half of the current generation of children will live in a singleparent household during some part of their childhood. Cuts in school budgets mean fewer adults per child. And declining neighborhood safety causes both youth and adults to keep more to themselves. What should society do? Clearly, we cannot abandon adolescents, especially young adolescents. While infants and toddlers are forming fundamental assumptions about human interactions, 10- to 14-year-olds are forming fundamental assumptions about society and their potential role in it. These assumptions are formed through observation of and interactions with adults and the adult world. If caring, concerned adults and role models are available to young people, they will be far more likely to develop into healthy, successful adults themselves (Furstenberg, 1993; Werner and Smith, 1992; Rutter, 1987; Garmezy, 1985). As the Carnegie Council on Adolescent Development’s report Great Transitions (1995) argues, the years of early adolescence— ages 10 to 14—are society’s last best shot at preventing social problems. With increased recognition of the growing number of adolescents who lack close adult attention, policy interest in mentoring as a form of social intervention has been advocated in such diverse areas as welfare reform, education, violence prevention, school-to-work transition and national service. The dramatic increase in the number of programs attempting to provide adult support for young people, particularly those in poverty, has occurred despite the absence of real evidence that such adult involvement can make a difference. Fittingly, it is a study of BBBS, arguably the bellwether of the mentoring movement, that provides the first such evidence. P/PV’s Mentoring Research This report is the centerpiece of P/PV’s eight-year research initiative to study mentoring. To place the findings in this report in context, we summarize our findings from other studies. Over the past eight years, P/PV has conducted a series of studies to explore the policy and operational implications of creating adult mentoring relationships for at-risk youth. We have examined the viability and effectiveness of several program models that embody the range of mentoring programs. This focus on existing programs was designed to inform wider, ongoing debate over social policy by tying the discussion to operational realities. The overarching questions the research initiative has addressed are: 1. Will participation in a mentoring program result in important, observable changes in the attitudes, perceptions and behaviors of at-risk young people? 2. What practices are required to administer mentoring programs effectively? What are the “best practices” regarding how much training, screening, matching and supervision to provide? 3. Is there a set of practices or features that characterize the adults who are effective in their mentoring relationships? Introduction 4. Are there large numbers of adults with the time and emotional resources to take on the demands of mentoring at-risk youth? 5. Can mentoring be integrated into large-scale youth-serving institutions, such as juvenile justice agencies? To provide credible evidence for answering these questions, we undertook several initiatives: an investigation of the Campus Partners in Learning program to study the usefulness of college students as mentors for middle school students at risk of academic failure; an assessment of the I Have A Dream tuitionguarantee and mentoring program at local affiliates in the Washington, D.C. area; an evaluation of the use of older citizens as mentors for at-risk youth in Temple University’s national Linking Lifetimes program; a study of mentoring demonstrations operated in Georgia and Missouri by the states’ juvenile justice systems; and, as the cornerstone of the research initiative, four studies of the content and effectiveness of the BBBS program. This report addresses our first research question by showing that participation in BBBS does lead to important, observable changes in the attitudes, perceptions and behaviors of at-risk youth. We speculate that other developmentally oriented mentoring programs that are similarly able to facilitate and carefully oversee long-lasting, intensive matches might have similar success. Our other studies show that the challenge for mentoring programs lies in strengthening their infrastructures and improving their program practices so that mentors and youth can meet long enough and consistently enough to form meaningful relationships (Tierney and Branch, 1992; Higgins et al., 1991). Although the recent mentoring movement emerged separately from BBBS, there is much that the mentoring field can learn from the practices of this pioneering one-to-one initiative (Furano et al., 1993). These reports begin to answer the question of what type of infrastructure is necessary to facilitate meaningful relationships—the second question in our research agenda. 3 A program’s infrastructure and support are critical in helping the adult and youth overcome the hurdles of forming a relationship and can help when obstacles arise during its course. To a large extent, however, it is the attitudes and actions of the volunteers themselves that lead to the creation of good relationships. Two studies (Morrow and Styles, 1995; Styles and Morrow, 1992) uncovered a set of adult practices that increase the chances that a mentor and youth will form a lasting, more mutually satisfying relationship—the third issue in our agenda. Three other reports addressed the fourth and fifth questions, concerning the feasibility of expanding and institutionalizing mentoring. In considering whether and how many more youth might be served, we found that more adults would be willing to mentor youth, but that many of these adults are not appropriate to the task (Roaf et al., 1994). Embedding mentoring in existing institutions and programs was found to be very difficult. The obstacles encountered in integrating mentoring into institutions are described by Greim (1992) and Mecartney et al. (1994). Organization of the Report Before presenting our findings on how BBBS improves the lives of the Little Brothers and Little Sisters, a number of characteristics about the program and the evaluation are described. Given the uniqueness of BBBS among mentoring programs, Chapter II lays out in detail the infrastructure and standards embedded in the BBBS program model, and describes the practices of the eight agencies that participated in this impact study. Chapter III describes the design of the evaluation. Chapter IV describes the characteristics of youth who participated in the study. Chapter V then presents the evidence on how youth who participated in a BBBS program differed, 18 months later, from similar youth randomly assigned to a control group. The final chapter summarizes the positive impacts of BBBS on youth, and draws policy implications for and about mentoring programs. 4 Making a Difference: An Impact Study of Big Brothers Big Sisters The Big Brothers Big Sisters Program The BBBS program has paired unrelated adult volunteers with youth from single-parent households for more than 90 years, using an approach that is intensive in delivery and broad in scope. Both the volunteer and the youth make a substantial time commitment, agreeing to meet two to four times per month for at least one year, with a typical meeting lasting four hours. BBBS is not a program targeted at ameliorating specific problems, but at developing the “whole person.” The relationship forged with a youth by the Big Brother or Big Sister creates the framework through which the mentor can support and aid the youth as he or she develops, traversing childhood and/or adolescence. who pose a safety risk, are unlikely to honor their time commitment or are unlikely to form positive relationships with the youth. (Refer to page 6 for a description of how these procedures are applied in the study agencies.) The application of the screening procedures is time-consuming and stringent. Earlier research found that after being under consideration for three to nine months, only 35 percent of applicants had been matched; 30 percent either withdrew or were considered by staff to be inappropriate, and 35 percent had not completed all the steps of the process (Roaf et al., 1994). Youth Screening A relationship between an unrelated adult and youth, the hallmark of the BBBS movement, is not established in a vacuum. Behind the hundreds of matches for which each agency is responsible is a professional staff with wide-ranging responsibilities. And undergirding the individual agencies are national operating standards that provide a level of uniformity in recruitment, screening, matching and supervision. While its standards are reinforced through national training, national and regional conferences, and periodic agency evaluations, BBBS is not monolithic. Individual agencies—including the eight agencies that participated in this study—adhere to national guidelines, but customize their programs to fit the circumstances of the cities and towns in which they are located. This chapter summarizes BBBS operational standards and implementational realities, and provides programmatic details about the eight study agencies. Operating Standards Working with over 500 local agencies, the BBBSA national office develops and publishes standards and required procedures to govern screening of volunteers and youth, orientation and training of the volunteer and the youth, and the creation and supervision of matches. These requirements represent minimum acceptable program practices—or the program irreducibles. Agencies can interpret them based on philosophy, geography, budget and the needs of the youth they serve, but these elements must be present. Volunteer Screening BBBSA’s most stringent guidelines concern procedures for screening volunteers. The purpose of the screening process is to protect the youth by identifying and screening out applicants The screening process for youth involves a written application, interviews with the parent and child, and a home assessment. Most agencies require that youth have no more than one parent/ guardian actively involved in their life, meaning that almost all youth deemed eligible live in single-parent households. Other youth eligibility criteria are age (from a minimum of 5 to a maximum of 18 years old), residence in the agency catchment area, a minimal level of social skills, and the agreement of the parent and child to follow agency rules. Training BBBS agencies provide an orientation for volunteers in which the program requirements and rules are explained. Many agencies also offer training on how to recognize and report incidents of sexual abuse. More extensive training is not required, but is recommended by the BBBSA office. Agencies that extend training generally include presentations on the developmental stages of youth, communication and limit-setting skills, tips on relationship-building, and recommendations on the best way to interact with a Little Brother or Little Sister. This information is designed to assist volunteers as they interact with their assigned youth, who are often from different racial or socioeconomic backgrounds. Matching and Meeting Requirements BBBSA says little about matching, other than recommending that agencies make matches based on each volunteer’s ability to help meet the needs of a specific youth. Yet a study of BBBS program practices found that agencies have developed remarkably similar matching criteria (Furano et al., 1993). In making matches, all the study agencies consider practical factors, such as gender, geographic proximity and availability. In addition, volunteers, youth and parents are asked to state their match The Big Brothers Big Sisters Program preferences. Volunteers indicate the type of youth they would like to be matched with, noting age, race and the types of activities they expect to engage in with the youth. Youth and their parents state their preference for volunteers, noting such factors as age, race and religion. Youth are asked about their activity preferences. One aspect of the process that differs across agencies is whether volunteers can choose the youth with whom they will be matched. While some agencies select and present the volunteer with a single youth, others allow the volunteer to choose from several youth. Although the parent/guardian of the youth must approve the selected volunteer, earlier research found that the parent/guardian rarely rejects a proposed volunteer (Furano et al., 1993). Supervision In an effort to facilitate effective matches, agencies emphasize supervision. National requirements specify that contact must be made with the parent, youth and volunteer within two weeks of the match. Monthly telephone contact with the volunteer is required during the first year of the match, as is monthly contact with the parent and/or youth. The youth must be contacted directly at least four times during the first year. Once the first year of the match has concluded, the requirement for caseworker contact with the participants is reduced to once per quarter. Case managers also support the match by providing guidance when problems arise in the relationship. BBBS and the Mentoring Field BBBS’s intensity and extensive infrastructure contrasts sharply with the laissez-faire structure of most of the newer programs. Part of the appeal of the initial wave of mentoring programs implemented during the 1980s was their seeming simplicity: advocates of these programs contended that adults could “naturally” work with youth. Mentors required only time and dedication, not screening, training or supervision. Founders of these programs recalled adults who served as their mentors— coaches, teachers and neighbors—and wanted to re-create that type of support with today’s youth. Thus, early recommendations for establishing and maintaining mentoring programs typically touted a laissez-faire approach that appealed to sponsors wary of instituting procedural and structural requirements they felt would intimidate volunteers. 5 A 1992 report by Marc Freedman warns of the danger of “fervor without infrastructure” in implementing mentoring programs: Merely hitching adults to kids, without adequate infrastructure, may create a sense of action, but is likely to accomplish little. It may even backfire. If a relationship engenders hurt or reinforces negative stereotypes, it is worse than no mentoring at all. P/PV’s previous mentoring research clearly points to the importance of volunteer screening and match supervision. We found that youth and mentors in programs with less infrastructure are less likely to meet, and therefore less likely to achieve a necessary condition for affecting the life of a youth: meeting long enough and with enough consistency to establish a relationship. BBBS matches are among the longest-lasting and most consistent (in terms of meeting) of any mentoring relationships. P/PV’s first study of BBBS found that 96 percent of first-year matches had met at least once in the previous four weeks and that, on average, the Big Brothers and Big Sisters had met with their Little Brothers or Little Sisters an average of 3.1 times during that period (Furano et al., 1993). By comparison, a study of six campus-based mentoring programs that served a population similar to that of BBBS, but had minimal volunteer screening, no criteria for matching and minimal supervision, showed a much lower rate of interaction. Only 57 percent of these matches met on a somewhat regular basis (Tierney and Branch, 1992). A study of two mentoring programs for youth in the juvenile justice system found that supervision in the two programs was limited, and the rate of interaction between the mentors and youth was correspondingly limited. Mentors in these programs missed more than a third of their scheduled weekly meetings. Among matches with non-incarcerated youth, only 40 percent of scheduled meetings took place (Mecartney et al., 1994). The only program we examined that came close to the meeting rate of BBBS was an intergenerational mentoring program that paired at-risk youth with elders. Sites for this program had screening, matching and supervision procedures, as well as paid mentors. Pairs met up to six times a month, a high rate that may have reflected the fact that the mentors were paid only if the meeting took place (Styles and Morrow, 1992). 6 Making a Difference: An Impact Study of Big Brothers Big Sisters Table 1 Characteristics of Study Agencies Characteristics Columbus Houston Total Active Matches 754 479 330 Race/Gender Minority Boys Minority Girls White Boys White Girls 14.5% 22.5 29.6 33.4 22.1% 32.2 27.3 18.4 Boys Ages 5-9 10-15 16+ 9.6% 29.3 5.2 Girls Ages 5-9 10-15 16+ 15.9% 35.5 4.5 Required Meetings Over First Year Required Length of Meeting Minneapolis Rochester Philadelphia Phoenix San Antonio Wichita 358 709 655 277 659 20.3% 29.1 20.0 30.6 13.7% 20.4 37.1 28.8 34.0% 21.6 29.5 14.9 10.5% 17.0 38.9 33.6 21.7% 31.4 23.8 23.1 18.1% 23.4 34.1 24.4 9.8% 35.3 4.4 3.6% 32.1 4.5 12.3% 29.0 9.5 7.7% 44.0 11.8 6.2% 40.0 3.2 5.0% 35.5 5.0 11.7% 31.6 8.9 10.2% 37.4 2.9 5.5% 44.9 9.4 9.8% 34.1 5.3 6.3% 23.8 6.4 13.9% 33.3 3.4 13.9% 35.1 5.5 13.5% 28.8 5.5 3x/Month 2-4x/Month 1/Week 1/Week 1/Week 1/Week 1/Week 1/Week 2-4 hours 3-4 hours 3-5 hours 3-5 hours 3-4 hours 3-6 hours 3-5 hours 3-4 hours Matches Lasting 12 Months or Longer 75% 75% 65-75% 70% 80% Cross-Race Matches 25% 33% 39% 47% 30% Agency Budget Number of Staff Full-time case managers Part-time case managers Note: All data are from 1992. n.a.= not available. n.a. 22% 64% 68% 34% n.a. $676,000 $998,000 $1,100,000 $505,000 $788,000 $848,000 $323,000 $802,000 24 10 5 26 13 1 29 6 5 10 4 2 23 11 0 25 15 0 12 6 0 31 8 13 The Big Brothers Big Sisters Program Study Agency Selection and Description From the network of more than 500 BBBS local agencies, we selected eight in which to study the effects of the program on youth. Agency participation was sought through presentations of the research agenda at BBBSA’s national conference, through an agency survey that requested a detailed profile of participants and program practices, and through in-person interviews with agency staff. The agencies that participated in the study were BBBS of Alamo Area (San Antonio, Texas), BBBS Association of Columbus and Franklin County (Columbus, Ohio), BB&S of Houston, BBBS of Greater Minneapolis, BBBS Association of Philadelphia, Community Partners for Youth (Rochester, New York), BB&S of Sedgewick County (Wichita, Kansas), and Valley BBBS (Phoenix, Arizona).3 The following were the key selection criteria for inclusion in the impact study: • A large active caseload and waiting list. So that the research effort would not reduce the number of youth served by the agency nor deny service to youth for substantially longer than would otherwise be the case, and to generate a sufficient number of youth for this study, chosen agencies had to have relatively large caseloads and waiting lists. • Geographic Diversity. The agencies were chosen for geographic diversity. Agencies were in the Northeast (Philadelphia and Rochester), the Midwest (Minneapolis, Columbus and Wichita), the South (Houston and San Antonio) and the Southwest (Phoenix). No agencies on the West Coast met the first two criteria (large waiting list and large active caseload) when sites were selected. The eight study agencies were among the largest in the BBBS federation, with an average active caseload of 528.4, 5 The total of 4,221 matches in the eight agencies represented approximately 6 percent of all BBBS matches during 1992. Table 1 shows that the study agencies served similar percentages of boys and girls. Only one agency served less than 40 percent of one gender (36.5% girls), which is explained by the presence of a nearby agency that made only Big Sister matches. The study agencies had annual budgets ranging from $323,000 to $1.1 million. Since Big Brothers and Big Sisters are unpaid, the majority of the agencies’ budget goes toward paying the professional staff who recruit, screen and train volunteers, and make and supervise the matches. In implementing the volunteer screening procedures, agencies required all applicants to submit a minimum of three written personal references and conducted a background investigation. This background investigation usually involved consulting the police records in the state in which the agency is located and attempting to identify volunteer applicants with a criminal history. Six of the eight study agencies also consulted the files of the state division of motor vehicles, with the intention of excluding volunteer applicants with dangerous driving records (e.g., multiple moving violations).6 Two of the eight study agencies submitted the volunteer applicants’ fingerprints to the FBI to search for past criminal activity. To identify potential child molesters, the agencies either administered a psychological test (half of our study agencies) and/or relied on an extensive in-person psychosocial interview. Five study agencies also visited volunteers’ homes to ascertain whether it would offer a safe environment for the Little Brother or Little Sister. The proportion of minority youth among those matched varied. Three agencies had caseloads that were over 50 percent minority youth; the lowest proportion among the agencies was 27.5 percent. This variability was due to a combination of varying racial composition in the communities and the difficulty some agencies have in recruiting a sufficient number of minority volunteers. Although agencies will match minority youth with white volunteers, many agencies and parents prefer to make same-race matches. The percentage of cross-race matches made by study agencies ranged from 22 percent to 47 percent. Tables with more detailed information about the study agencies can be found in Appendix B. 7 8 Making a Difference: An Impact Study of Big Brothers Big Sisters Research Design This chapter describes the basic research design. It first discusses the hypothesized impacts of participation in the program, then details the random assignment methodology used to test for the presence of these impacts. 4. Improved Self-Concept. A successful relationship might affect how a Little Brother or Little Sister feels about himself or herself. Therefore, we hypothesized that program youth would report a better sense of competence and self-worth than their non-program counterparts. Hypothesized Impacts 5. Social and Cultural Enrichment. Many of the activities that the volunteer and youth participate in over the course of a match expose the Little Brother or Little Sister to new experiences. Therefore, we hypothesized that Little Brothers and Little Sisters would report taking part in more activities, such as attending sporting events or going to a library, than their non-program counterparts. The first task was to identify the appropriate impacts to measure in the context of the BBBS program. We developed our list of potential impacts working closely with staff from the BBBSA national office; with the local agencies; and through a review of BBBSA’s manual of standards and practices. The national manual lists five “common” goals for a Little Brother or Little Sister: developing a successful relationship; providing social, cultural and recreational enrichment; improving peer relationships; improving self-concept; and improving motivation, attitude and achievement related to schoolwork. In addition, conversations with BBBS staff suggested that having a Big Brother or Big Sister could reduce the incidence of antisocial behaviors, such as drug and alcohol use, and could improve a Little Brother or Little Sister’s relationship with the parent. We thus hypothesized that participation in BBBS would result in some or all of the following impacts: 1. Reduced Antisocial Activities. By providing youth with good role models, and helping them cope with peer pressures, think through the consequences of their actions and become involved in socially acceptable activities, volunteers would inhibit youth from initiating alcohol or drug use, and delinquent behavior. 2. Improved Academic Outcomes. By showing that they value education, taking an interest in the youth’s school progress, and stressing the importance of education to later success, volunteers might influence their Little Brothers’ and Little Sisters’ attitudes toward school and their school performance. Therefore, we hypothesized that Little Brothers and Little Sisters would value school more, have better attendance and perhaps even get better grades. 3. Better Relationships with Family and Friends. The volunteer can help the youth learn how to trust others, express negative feelings more productively, and generally become more able to relate effectively with others. Therefore, we hypothesized that: (1) BBBS participation would have positive effects on the youth’s relationship with their custodial parent (usually their mother); and (2) participation in BBBS would have positive effects on the youth’s relationships with their peers. Developing a successful relationship, a goal listed in the BBBSA manual of standards and practices, is not included as a hypothesized impact. We view the development of a successful relationship as the core of the program treatment rather than an outcome of participation. Developing a successful relationship is an important mediating factor and earlier research has extensively described how a successful relationship develops.7 (See Morrow and Styles, 1995.) Design Strategy The effect of having a Big Brother or Big Sister on the life of a youth was determined in this evaluation by studying two randomly assigned groups of 10- to 16-year-olds who applied to the study agencies during the intake period.8 One group of applicants, the randomly selected control group, was put on the waiting list for a Big Brother or Big Sister for 18 months; case managers attempted to match the other randomly selected group—i.e., the treatment group—as quickly as possible. The two groups were then compared at follow-up. The Reason for Random Assignment Use of a classical experimental methodology with random assignment to either a treatment or control group was the only way to reach definitive conclusions about the impact of participation in the BBBS program. This random assignment design ensures that the treatment and control groups are statistically equivalent, on average, with respect to all characteristics except program participation. How does random assignment do this? While two randomly chosen individuals are unlikely to be the same age, the average age of two fairly large groups of people randomly selected from the same population is likely to be quite close. In fact, the average of all characteristics of these two large groups is likely to be quite similar. Thus, if the average behavior of the two groups (treatments and controls) differs after the intervention, the difference can be confidently and causally Research Design linked to participation in the program. Hence, the strength of a random assignment design is that the outcomes exhibited by the control group accurately approximate what would have happened to treatment group members if they had not received the intervention. Some consider random assignment unethical because it denies services to control group youth. While our research design had to include a waiting period for control youth, we addressed such ethical concerns by: (1) ensuring that the total number of matches made by an agency did not decline, and (2) using a follow-up period (18 months) that, in many cases, was no longer than an agency’s usual waiting period. During the study period, agency staff processed twice the usual number of youth—50 percent of whom were assigned to the treatment group and eligible to be matched with a Big Brother or Big Sister, and 50 percent of whom were assigned to the control group. Before the study began, the average waiting period at the study agencies for boys often exceeded 18 months; the waiting period for girls, while substantially less, still ranged from three to 20 months. 9 could not abrogate. Across the study agencies, 61 youth were excluded because they were being served under a contractual obligation. The random assignment process consisted of three major steps: 1. Through either a personal interview or group presentation, agency staff explained the research project to youth and their parent or guardian, and obtained the consent of both for youth to participate in the research.9 2. Agency staff reviewed each application where consent was obtained and determined whether the youth was eligible for the program using their usual procedures. 3. Once a youth was determined to be eligible, P/PV’s survey subcontractor randomly assigned him/her to either the treatment or control group. Although individual agencies tailored processing procedures to fit their own operations, no youth were randomly assigned until agency staff deemed them eligible for the program, and both they and their parents had consented to participation in the research. Implementation of Random Assignment All age-eligible youth who came to the study agencies during the research intake period were required to participate in the intake procedures. There were three exceptions to this requirement: • A youth was excluded if he/she could not complete a telephone interview. Youth fitting this description included those with severe physical or learning disabilities. Families without telephones were included in the research; they called the survey firm’s toll-free number from a friend’s house or the BBBS agency’s office. Across the study agencies, 13 youth were excluded because they could not complete a telephone interview. • Youth who were not a part of the BBBS core program were excluded. Across the study agencies, approximately 50 youth were excluded because they were in a special program, such as the Native American program at Valley Big Brothers Big Sisters in Phoenix. In addition, two agencies ran satellite programs at local colleges. While participants in these programs were official BBBS participants, the program operated under different guidelines; thus, including them in the research would have been analogous to evaluating a college mentoring program rather than BBBS’s core program. • Youth being served under a contractual obligation were excluded. Two agencies had agreements with their local child protective services; another agency had an agreement with two youth-serving organizations that the research In explaining the study to parents and youth, staff pointed out that because youth in the treatment group would receive priority for matching, youth who agreed to participate would have a 50 percent chance of being matched more quickly. Parents also understood that their child had a 50 percent chance of being assigned to the control group, which would mean waiting 18 months before the agency would resume processing their application. If a parent or youth refused to participate in the research study, the agency placed the youth on the waiting list for 12 months. Only 32 youth and/or parents (2.7%) at these agencies refused to participate in the research. After they determined that a youth was eligible for the program and the parent/guardian and youth signed a consent form indicating that they understood the study, agency staff submitted the name of the youth to P/PV’s survey subcontractor for assignment. Sample Intake Sample intake ran from October 1991 to February 1993. Agencies were required to implement the random assignment procedures until they reached their sample size goal or until February 1993, whichever came first. Based primarily on the size of their existing caseloads, agencies were assigned varying sample size goals— two agencies had a goal of 230, five a goal of 150 and one a goal of 80. Ultimately 1,138 youth from eight agencies were enrolled in the study over a 17-month period. 10 Making a Difference: An Impact Study of Big Brothers Big Sisters Matching Treatment Youth A major goal of the research design was to minimize the design’s interference in the matching process while maximizing the number of treatment youth who were matched. To achieve these potentially conflicting goals, we directed case managers not to modify their usual matching criteria, but to prioritize the matching of treatment youth when similar youth were being considered for a specific volunteer. For example, when a case manager determined that a volunteer would work equally well with a 9-year-old girl who was not a part of the evaluation and an 11-year-old treatment group girl from the same area, we instructed the case manager to match the 11-year-old. Data Sources Reaching conclusive statements about whether having a Big Brother or Big Sister makes a difference in the life of a young person required information from the youth, parent and case manager at three critical times—at baseline, at the time of the match, and at follow-up. We accomplished this by: • Administering two surveys to the parent/guardian and the youth (one at the time of random assignment and one 18 months later); • Asking case managers to complete four data collection forms—two when the study was explained to potential participants, and one each at the time the match was made and 18 months after random assignment; • Asking a key informant to provide background information about the agency and its program practices.10 The centerpieces of data collection were the baseline and followup interviews with sample members and their parent/guardian. The baseline interviews occurred immediately after random assignment but before sample members were told whether they were in the treatment or control group. During the baseline interview, the parent was asked to provide general background information, such as his/her years of completed education, welfare receipt by any household members, labor force status and relationship to the youth. The interviewers asked the youth to provide basic demographic information (e.g., age, race/ethnicity, family structure), information on services other than a match that they may have participated in through BBBS, and baseline measures for the outcome variables. Follow-up interviews were conducted 18 months after random assignment for every sample member who completed a baseline interview. Parents were asked to evaluate the performance of the volunteer, their satisfaction with the BBBS agency and whether they thought the program had made a difference in their child’s life, as well as to answer questions about their labor force status and household income. Interviewers asked youth to provide the follow-up measures of the outcome variables, and for the treatment youth, they asked about their relationship with their Big Brother or Big Sister. Table 2 shows how the sample evolved to the final analysis sample. From October 1991 through February 1993, 1,138 youth were randomly assigned to either the treatment or control group, with 1,107 (97.3%) completing a baseline interview. From April 1993 to September 1994, follow-up interviews were attempted with 1,107 youth; interviewers completed 959. (See Appendix A for a fuller discussion of the interviewing process.) The final response rate of almost 85 percent exceeds acceptable research standards for this type of survey. For both treatments and controls, case managers were asked to complete two forms when the parent and youth were given the opportunity to participate in the research. The first, the client data form, collected basic information about each youth, and was designed to determine whether the youth was eligible for the study by securing consent for participation, and ascertaining their age and their ability to speak English or Spanish sufficiently well to complete an interview. Information (name, address and telephone number of youth) that allowed the interviewers to administer the baseline survey was also gathered. The research sample form, the second form completed by case managers, provided detailed background information on the youth and his/her family. This form asked for information about the gender and age of the parent and family structure, and included a series of deeply personal questions about the youth, including whether the case manager believed the young person had been the victim of sexual, physical or emotional abuse, or had any physical or learning disabilities. Also on the form was whether the family had a history of substance abuse or domestic violence, and how the case manager anticipated that the youth would benefit from participation in BBBS. The match form was completed by the case managers when the Little Brother or Little Sister was assigned to a volunteer. This form served two purposes—it provided information about the volunteers (e.g., age, gender, years of completed education, income, occupation) and allowed us to monitor when matches were taking place. 11 Research Design The final form—the follow-up form—was completed 18 months after random assignment; it provided detailed information about the case manager’s perception of the volunteer’s performance, a description of problems (if any) that occurred during the match, the reason for terminating the match (if applicable), and several questions about the match itself, including the length and frequency with which the pair met and the goals for the match. For treatment youth who were never matched, the case manager recorded the reason that the agency was unable to make a match. The final component of the data collection strategy was gathering information that allowed us to describe the agencies themselves, including their individual program practices and information about the type of youth that each served. In 1992, we asked a senior staff member in each site to complete a survey with a wide-ranging series of questions. All eight agencies completed the survey, which provided us with the age, race and gender of all youth served by an agency, their volunteer screening and training procedures, and match supervision guidelines. Table 2 Sample Composition Treatment Control Total Number of Youth Randomly Assigned 571 567 1,138 Number of Youth with Baseline Surveys 554 (97.0%) 553 (97.5%) 1,107 (97.3%) Number of Youth in the Analysis Sample 487 (85.3%) 472 (83.2%) 959 (84.3%) 12 Making a Difference: An Impact Study of Big Brothers Big Sisters The Sample Youth and the Volunteers This chapter describes the young people in the analysis sample, and the Big Brothers and Big Sisters who were matched with youth in the sample. terms. BBBS agencies match only within gender and try to make same-race matches. In this section, we discuss baseline characteristics for the sample as a whole, except when there are large subgroup differences. Background Characteristics of Study Sample Youth Table 3 shows the race/gender and age for the youth in the analysis sample (487 treatments and 472 controls). Just over 60 percent of the sample were boys (62.4%), and over 55 percent were members of a minority group. At about 15 percent, white girls were the smallest subgroup, and at about 34 percent, minority boys were the largest. Seventy-one percent of the minority youth were African American, 18 percent were Hispanic, 5 percent were biracial, 3 percent were Native American and 3 percent were members of a variety of other racial/ethnic groups. Sixty-nine percent of youth came to the program between the ages of 11 and 13. In this section, we describe the baseline characteristics of the youth in the study sample, and their households. Since no meaningful differences in the baseline characteristics of the treatments and controls emerged—a byproduct of random assignment that was confirmed by statistical analysis (Appendix A)—we do not differentiate between treatments and controls when discussing the background characteristics of the youth, except in Table 3, which presents the age, race and gender of the analysis sample. The tables contain information for the sample as a whole and for six subgroups: boys, girls, minority boys, minority girls, white boys and white girls. We examine these subgroups partly because the BBBS agencies think of their caseload in these Table 4 shows that about 90 percent of the youth lived with only one of their parents, and another 5.6 percent lived with only one of their grandparents. Living with a grandparent was slightly more common among minority youth. About 20 percent of these parents/guardians did not graduate from high Table 3 Race/Gender and Age of Youth by Treatment Status Characteristics Treatments Controls Overall Race/Gender Minority Girls White Girls Minority Boys White Boys 21.8% 15.6 33.1 29.4 23.6% 14.0 35.1 27.2 22.7% 14.9 34.1 28.3 Age at Baseline 10 11 12 13 14 15 16 10.1% 24.4 25.5 20.1 13.1 5.5 1.2 10.8% 24.4 22.3 21.2 15.0 5.3 1.1 10.4% 24.4 23.9 20.7 14.1 5.4 1.2 Number of Youth 487 472a 959 a Three youth did not report their race; thus, the number of youth assigned to the four race/gender groups is 956. 13 The Sample Youth and the Volunteers Table 4 Characteristics of the Study Youth’s Households and Parents/Guardians Characteristics Total Boys Girls Minority Girls White Girls Minority Boys White Boys Parent/Guardian Relationship to Client Parent Foster parent Grandparent Aunt/Uncle Guardian Other 90.2% 1.3 5.6 2.0 0.1 0.9 91.3% 1.0 5.0 2.2 0.2 0.3 88.2% 1.7 6.4 1.7 0.0 2.0 84.6% 1.9 8.4 2.3 0.0 2.8 93.7% 1.4 3.5 0.7 0.0 0.7 88.6% 1.2 6.5 3.4 0.0 0.3 94.5% 0.8 3.3 0.8 0.4 0.4 Parent/Guardian Level of Education Less than high school graduate High school diploma/GED Vocational/Technical Some college Associate’s degree College degree or more 21.6% 36.3 4.6 25.9 3.8 7.8 18.6% 37.0 4.7 26.6 3.8 9.4 26.8% 35.2 4.5 24.8 3.7 5.1 34.0% 32.1 4.3 24.1 2.8 2.8 16.2% 40.1 4.9 25.4 4.9 8.5 21.2% 36.3 5.5 25.2 3.1 8.6 15.5% 37.6 3.7 28.0 4.8 10.3 Youth Living in Households Receiving Public Assistance 43.3% 37.1% 53.5% 62.6% 40.1% 45.8% 27.0% Household Income Less than $10,000 $10,000 to $24,999 $25,000 to $39,999 $40,000 to $54,999 $55,000 or more 39.7% 43.0 13.1 3.3 1.0 34.3% 45.8 15.2 4.2 0.5 49.0% 38.2 9.3 1.8 1.8 60.1% 30.1 7.9 1.5 0.5 33.1% 49.6 11.5 2.2 3.6 44.6% 39.9 13.1 2.5 0.0 21.9% 52.8 17.8 6.3 1.1 Number of Youth 959 599 360 217 142 326 271 Note: Three youth did not report their race; thus, the number of youth assigned to the four race/gender groups is 956. 14 Making a Difference: An Impact Study of Big Brothers Big Sisters Table 5 Stressful Life Experiences of the Youth Characteristics Total Boys Girls Minority Girls White Girls Minority Boys White Boys Youth Experiencing: Death of a parent/guardian Divorce or separation of parent/guardian Serious illness/injury of youth or significant other Arrest of youth or significant other Family history of substance abuse Family history of domestic violence Significant physical disability Significant learning disability Significant health problems 14.6% 39.9 15.9% 40.0 12.5% 40.0 14.3% 29.5 9.9% 56.3 13.2% 30.8 18.8% 50.9 6.1 7.1 40.3 28.3 2.9 15.6 9.0 9.0 6.0 41.5 28.1 2.9 18.3 9.8 6.1 8.9 38.3 28.6 3.1 11.2 7.8 3.7 10.1 36.9 26.3 1.4 7.9 7.4 9.9 7.0 40.9 32.4 5.6 16.2 8.5 7.7 4.0 33.2 23.7 2.5 14.2 9.6 10.7 8.1 51.9 33.7 3.3 22.9 10.0 Youth Experiencing Physical, Emotional or Sexual Abuse (reported by case manager): Any form of abusea Physical abuse Emotional abuse Sexual abuse 27.1% 11.2 21.3 7.3 26.3% 11.5 21.2 4.9 28.6% 10.6 21.4 11.4 22.1% 9.2 16.1 8.8 38.7% 12.7 29.6 15.5 19.4% 10.5 14.2 2.8 34.7% 12.9 29.9 7.4 Number of Youth 959 599 360 217 142 326 271 Note: Three youth did not report their race; thus, the number of youth assigned to the four race/gender groups is 956. a Some youth had suffered multiple forms of abuse. 15 The Sample Youth and the Volunteers Table 6 Characteristics of Never-Matched Treatment Youth Characteristics Total Boys Girls Minority Girls White Girls Minority Boys White Boys Reason Youth Not Matcheda No suitable volunteer found Youth no longer wished to be matched Youth no longer suitable for BBBS Family structure changed Youth moved out of area 19.3% 28.4 10.1 10.1 10.1 19.1% 26.2 8.3 10.7 7.1 20.0% 36.0 16.0 8.0 20.0 18.8% 31.3 18.8 6.3 12.5 22.2% 44.4 11.1 11.1 33.3 13.0% 26.1 10.9 10.9 4.4 27.0% 27.0 5.4 10.8 10.8 109 (22.4%) 84 (27.5%) 25 (13.7%) 16 (15.1%) 9 (11.8%) 46 (28.6%) 37 (25.9%) Total Number of Never-Matched Youth Note: One boy did not report race. a Case manager could check multiple items. school, and over 35 percent had completed only high school or earned a GED. About 25 percent of the parents/guardians had some college experience. Many of the youth lived in poor households—over 40 percent were receiving either food stamps and/or cash public assistance. Minority girls were the most likely to live in homes collecting welfare (62.6%), while white boys were the least likely (27.0%). Minority boys and white girls were about equally likely to live in homes receiving public assistance. As shown in Table 5, a significant number of study sample youth had experienced difficult personal situations, such as the divorce or separation of their parents, a family history of substance abuse or domestic violence, or being the victims of physical, emotional and/or sexual abuse. Approximately half of the white youth and one-third of the minority youth had experienced the divorce or separation of their parents/guardians. Fifteen percent of the youth had experienced the death of a parent/guardian. Over 25 percent of the youth lived in homes with a history of domestic violence and 40 percent resided in homes with a history of substance abuse; both these experiences were more characteristic of white than of minority youth’s households. More than one-quarter of the youth had experienced either physical, emotional or sexual abuse. White youth were more likely than were minority youth to have experienced some form of abuse. The most prevalent form of abuse was emotional abuse, experienced by approximately 30 percent of the white youth and 15 percent of the minority youth. White girls were the most likely to be victims of sexual abuse (15.5%). 16 Making a Difference: An Impact Study of Big Brothers Big Sisters The Treatment Youth The Volunteers The only difference between the treatment and control group youth was that the treatment youth had the opportunity to be matched with a Big Brother or Big Sister.11 This section discusses how matches were made in the context of the evaluation, how long it took to match the treatment youth, how long they were matched, and why some treatment youth were not matched during the study period. During the study period, 409 Big Brothers and Big Sisters were paired with treatment youth.12 The average age of the 236 men who were matched with Little Brothers in the study sample was 30; the average age of the 173 women was 28. Selecting an appropriate volunteer to match with a youth is perhaps the most important program practice. Agency staff decisions to pair an adult volunteer with a specific Little Brother or Little Sister are affected by a variety of factors—among them, shared interests, reasonable geographic proximity, preferences for same-race matches, and a desire to match youth who have been waiting the longest. At the conclusion of the study period, 378 (78%) of the treatment youth in the analysis sample had been matched. About 90 percent of the girls and 75 percent of the boys had been matched. This gender differential is consistent with the typical experience of BBBS agencies, which have historically had difficulty recruiting sufficient male volunteers to meet the demand for Big Brothers. As shown in Table 6, agency staff reported three major reasons for the failure to match 109 treatment youth during the study period: • Thirty-three of the unmatched treatment youth became ineligible for BBBS matches during the study period. These changes in status, which occurred after random assignment but before a match could be made, were due to such events as the parent remarrying, or the youth getting too old or changing place of residence. • Thirty-one were not matched because the youth did not want or no longer wanted a Big Brother or Big Sister. Agency staff reported that some parents will request a Big Brother or Big Sister for a child who does not want one. If a case manager determines that this is the case, he/she will not make a match. • Twenty-one were not matched because a suitable volunteer could not be found during the study period. Agency staff will not make a match solely for the sake of making a match. Even though staff were prioritizing the matching of treatment youth, they would rather not make a match than make a bad one. • The 24 remaining treatment youth were not matched for a variety of reasons, most often because the parent or youth did not follow through with the intake process. As shown in Table 7, the Big Brothers and Big Sisters were generally well-educated young professionals. Only 13 percent had a high school education or less, and more than 60 percent had a college or graduate degree. Nearly half worked in professional or managerial positions, another one-quarter held technical, sales or administrative jobs, and about 10 percent were students. Only one-third lived in households with less than $25,000 in income, and almost 30 percent lived in homes with incomes of $40,000 and over. About three-quarters were white, which resulted in approximately 60 percent of the minority youth being matched with a white Big Brother or Big Sister. BBBS agencies will match a Big Brother or Big Sister with a second Little Brother or Little Sister when their first previous match ends, provided that the reason the match ended was not due to the volunteer’s inability to engage in a successful match. Among the volunteers matched with Little Brothers or Little Sisters in the study sample, over 10 percent had previously served as a Big Brother or Big Sister. Length of Matches How long a treatment youth had been meeting with the Big Brother or Big Sister at the conclusion of the study period depended on how long it took the agency to find an appropriate volunteer and how long the match itself lasted. Table 8 shows that on average, agencies needed six months to match minority boys, five months to match white boys, almost four months for minority girls, and three and a half months for white girls. At the time of the follow-up interview, the average length of match for treatments who had been matched was almost 12 months, with white girls having met with a Big Sister for the longest period (12.3 months) and minority boys having met with a Big Brother for the shortest (10.7 months). Little Brothers and Little Sisters met with their Big Brothers and Big Sisters on a regular basis. Over 70 percent of the youth met with their Big Brother or Big Sister at least three times a month, and approximately 45 percent met one or more times per week. At the time of the follow-up interview, 229 of the 378 matched treatment youth were still meeting with their Big Brother or Big Sister, while 149 treatment youth were no longer matched. The Sample Youth and the Volunteers Table 7 Demographic Characteristics of Volunteers by Gender Big Brothers Big Sisters Age 16-19 20-24 25-29 30-34 35-39 40 + 1.7% 22.8 37.8 16.3 6.9 14.6 1.2% 38.0 31.6 13.5 6.4 9.4 Race White Minority 71.9% 28.1 75.4% 24.6 19.0% 13.7% 4.6% 18.4 40.1 19.8 17.0 5.1% 42.4 34.8 12.0 5.7 Completed Years of Education High School Diploma or Equivalent Some College College Graduate Graduate Education 11.1% 24.4 50.4 14.1 14.6% 29.8 43.3 12.3 Occupation Unemployed Student Retired Managerial/Professional Technical/Sales/Administrative Service Other 0.4% 8.3 0.4 51.7 23.5 10.0 5.6 0.0% 13.7 0.6 44.6 30.4 7.7 3.0 Previously Served as Big Brother or Big Sister 13.2% 11.1% Number of Volunteers 236 173 Have Own Children Household Income < $10,000 $10,000 - 24,999 $25,000 - 39,999 $40,000 - 54,999 $55,000 + Note: 19 men and 14 women did not answer the household income question. On the remaining questions, each group had less than 10 missing responses per item. 17 18 Making a Difference: An Impact Study of Big Brothers Big Sisters Table 8 Characteristics of the Matches Characteristics Total Boys Girls Minority Girls White Girls Minority Boys White Boys Time to Match and Length of Match by Gender and Race Average time to match (months) Average total exposurea (months) 4.7 11.4 5.4 10.9 3.6 12.0 3.9 11.8 3.4 12.3 5.9 10.7 4.9 11.2 How Often Little Brother or Little Sister Met With Big Brother or Big Sister Two times per week Once a week Three times per month Two times per month Once per month 4.5% 41.7 24.4 24.2 5.3 5.8% 41.2 22.6 25.2 5.3 2.6% 42.6 27.1 22.6 5.2 2.2% 39.6 29.7 24.2 4.4 3.1% 46.9 23.4 20.3 6.3 4.2% 35.8 27.5 24.2 8.3 7.6% 47.2 17.0 26.4 1.9 Number of Matched Youth 378 221 157 90 67 115 106 a Combined length of all matches, including closed first matches and those still meeting at time of follow-up interview. The figure is based only on the ever-matched sample. The Sample Youth and the Volunteers Summary This chapter highlighted several key data: • Of the 959 youth in the sample, almost 60 percent were minority youth and over 60 percent were boys. Many were poor, with 40 percent living in homes receiving public assistance. A substantial number had experienced disruptive personal circumstances: 40 percent lived in families with a history of substance abuse, 28 percent in families with a history of domestic violence, and 27 percent were themselves the victims of emotional, physical or sexual abuse. • Over 400 volunteers were matched with study sample youth. These Big Brothers and Big Sisters were generally welleducated young professionals. About 60 percent were college graduates, while only 13 percent had earned just a high school degree or GED. About two-thirds lived in homes where the total income of all household members was greater than $25,000, with about 40 percent living in homes with over $40,000 in income. About 50 percent held managerial or professional positions, and 25 percent held technical, sales or administrative jobs. • Of the 487 youth in the treatment group, 378 (78%) were matched with a Big Brother or Big Sister during the study period; on average, youth were matched with a Big Brother or Big Sister for 12 months during that period. About 70 percent of the matches met three or four times a month, with an average meeting lasting four hours. The following chapter presents findings on whether participation in a BBBS program made a difference in the lives of Little Brothers and Little Sisters. 19 20 Making a Difference: An Impact Study of Big Brothers Big Sisters The Impact on Youth of Having a Big Brother or Big Sister Mentoring programs that pair adults with young people have been hypothesized to have multiple benefits for the youth. In this chapter, we present evidence concerning the benefits of participation in the BBBS program. We measured program impacts 18 months after a youth was deemed eligible to participate in a BBBS program, with the expectation that this period would give agency staff sufficient time to find a suitable volunteer for the youth and give the match sufficient time to develop and begin to affect the youth. Identifying an appropriate set of outcomes to determine whether participation in a BBBS program makes a difference in the life of a youth is a complex task, particularly since BBBS is an individualized program with different goals for each match. As discussed in Chapter III, we selected the following set of outcome areas: • Antisocial Activities; • Academic Performance, Attitudes and Behaviors; • Relationships with Family; The 959 youth in the study sample (487 treatments and 472 controls) came to the program when they were, on average, 12 years old. Almost 60 percent were members of a minority group, and over 60 percent were boys. The vast majority (over 80%) came from relatively poor households. Almost 80 percent of the treatment youth were matched with a Big Brother or Big Sister during the study period; on average, the relationships had lasted almost one year at the conclusion of the study period (i.e., the time of the follow-up survey). • Relationships with Friends; • Self-Concept; and • Social and Cultural Enrichment. Although improvements in each of these areas are not explicit goals for every match, they are the objectives most frequently cited by BBBS staff. The program might have had effects on other outcomes that we did not measure. Table 9 Net Impact of Participation in BBBS on Initiating Use of Drugs and Alcohol Change in the Likelihood of Initiating Drug Abuse Net Impact Follow-up Control Mean Change in the Likelihood of Initiating Alcohol Use Net Impact Follow-up Control Mean Overall -45.8%** 11.47% -27.4%* 26.72% Gender Male Female -55.0%** -26.6 11.54% 11.36 -19.2% -38.8 26.48% 27.08 Race/Gender Minority Male Minority Female White Male White Female -67.8%** -72.6* -32.7 49.5 13.41% 11.50 9.09 11.29 -11.4% -53.7* -34.5 -8.4 21.60% 26.97 33.33 27.78 Note: The size of the analysis sample was 959: 217 minority girls, 142 white girls, 326 minority boys, 271 white boys, and 3 youth who did not give their race. ** Indicates that the impact differs statistically from zero at the 0.05 level. * Indicates that the impact differs statistically from zero at the 0.10 level. 21 The Impact on Youth of Having a Big Brother or Big Sister Table 10 Net Impact of Participation in BBBS on Hitting, Stealing and Damaging Property Number of Times Hit Someone Net Impact Overall -.85** Follow-up Control Mean Number of Times Stole Something Net Impact Number of Times Damaged Property Follow-up Control Mean Net Impact Follow-up Control Mean 2.68 -.05 .26 -.03 .20 Gender Male Female -.67 -1.17* 2.67 2.69 -.07 -.02 .27 .24 -.04 -.03 .24 .13 Race/Gender Minority Male Minority Female White Male White Female -.09 -1.45 -1.54* -.37 2.13 3.04 3.39 1.85 .01 -.07 -.16 .06 .24 .27 .30 .20 .02 -.02 -.10 -.05 .30 .13 .16 .14 Note: The size of the analysis sample was 959: 217 minority girls, 142 white girls, 326 minority boys, 271 white boys, and 3 youth who did not give their race. ** Indicates that the impact differs statistically from zero at the 0.05 level. * Indicates that the impact differs statistically from zero at the 0.10 level. In the following sections, we discuss impacts in each of these six outcome groups. We considered between four and 10 outcomes for each group. Outcome variables were classified as either attitudinal or behavioral. The attitudinal outcomes were typically scales measured by a series of items or questions combined to form a single measure. The behavioral outcomes were typically based on the responses to single questions—e.g., How often were you sent to the principal’s office? How many hours per week do you spend doing homework?14 All outcome variables we considered are listed in Appendix A, which also includes the reliability analysis for the attitudinal scales. The impact estimates presented here represent a comparison of the average experience of treatment group members with the average experience of control group members.15 Overall impact estimates were calculated by comparing all treatments to all controls. A negative net impact indicates that the treatment value is lower than the control value; a positive net impact indicates that the treatment value is higher than the control value. Subgroup impacts compare the treatment youth in that subgroup with the control youth in the same subgroup. The experience of the control group represents what would have happened to the treatment group had they not been given the opportunity to participate in the BBBS program. Any differences that develop between the two groups can be confidently attributed to a youth’s participation in the BBBS program.16 For ease of presentation, we refer to the treatment group as “Little Brothers and Little Sisters,” even though this group includes some treatment youth who were never matched. We highlight only impacts that are statistically significant at a .10 level of confidence. 22 Making a Difference: An Impact Study of Big Brothers Big Sisters Antisocial Behaviors We hypothesized that the relationships youth experience in BBBS would lead them to exhibit fewer antisocial behaviors, as suggested by Furstenberg (1993) and Werner and Smith (1992). The two most important antisocial behaviors we considered were the initiation of drug and alcohol use. Elliot (1993) presents evidence that delaying the onset of the use of illegal drugs and alcohol decreases the likelihood that the youth will engage in problem behaviors, such as criminal activity and school failure. Some might argue that it is less important to delay the onset of alcohol use, since most teens experiment with alcohol at some point. However, Elliot reports that among youth who never use alcohol, the risk of serious delinquency is reduced by a factor of four. Thus, delaying alcohol use should decrease the likelihood of delinquency. As shown in Table 9, we found that Little Brothers and Little Sisters were significantly less likely than their control counterparts to start using illegal drugs and alcohol during the study period. Little Brothers and Little Sisters were 45.8 percent less likely to start using illegal drugs than were their control counterparts. The impact was largest among minority Little Brothers and minority Little Sisters, both of whom were approximately 70 percent less likely than their control counterparts to have started using illegal drugs. Put differently, for every 100 minority boys in this age group who start using illegal drugs, only 33 similar minority boys who have a Big Brother will start using illegal drugs. For every 100 minority girls in this age group who start using illegal drugs, only 28 similar girls who have a Big Sister will start using illegal drugs.17 The results for initiating alcohol use were not as large as those for initiating drug use, but were still impressive: Little Brothers and Little Sisters were 27.4 percent less likely than control Table 11 Net Impact of Participation in BBBS on Academic Outcomes Perceived Ability to Complete Schoolwork (Scholastic Competence) Net Impact Follow-up Control Mean Grade Point Average (GPA) Net Impact Follow-up Control Mean Number of Times Skipped Class Net Impact Follow-up Control Mean Number of Times Skipped a Day of School Net Impact Follow-up Control Mean Overall .71*** 16.36 .08* 2.63 -.51** 1.39 -.47*** .90 Gender Male Female .39 1.25*** 16.64 15.89 .03 .17** 2.60 2.67 ## -.18 -1.07*** 1.05 1.95 ### -.02 -1.22*** .57 1.45 Race/Gender Minority Male Minority Female White Male White Female -.11 1.52*** 1.06** .81 17.11 15.67 16.05 16.27 .06 .20* .01 .10 2.58 2.62 2.63 2.74 -.27 -.92** -.10 -1.36** 1.25 2.01 0.81 1.88 ### .22 -.98*** -.31 -1.66*** 0.51 1.26 0.66 1.80 Note: *** ** * ### ## The size of the analysis sample was 959: 217 minority girls, 142 white girls, 326 minority boys, 271 white boys, and 3 youth who did not give their race. Indicates that the impact differs statistically from zero at the 0.01 level. Indicates that the impact differs statistically from zero at the 0.05 level. Indicates that the impact differs statistically from zero at the 0.10 level. Indicates that the impact was not the same across subgroups at a 0.01 level of significance. Indicates that the impact was not the same across subgroups at a 0.05 level of significance. The Impact on Youth of Having a Big Brother or Big Sister Academic Attitudes, Behavior and Performance youth to start using alcohol. The impact was greatest among the minority Little Sisters, who were less than half as likely to start drinking alcohol. Put differently, for every 100 minority girls in this age group who start to use alcohol, only 46 similar girls who have a Big Sister will start using alcohol. As Table 11 shows, we found that Little Brothers and Little Sisters earned higher grades, skipped fewer classes and fewer days of school, and felt more competent about doing their schoolwork than did control youth. The impacts were larger for girls. We looked at a number of other indicators of antisocial behavior. Table 10 shows the most important of these: how often the youth hit someone, stole or damaged property over the past year. While we did not find any impacts on the number of times a youth stole or damaged property, Little Brothers and Little Sisters were 32 percent less likely to report hitting someone during the previous 12 months.18 We also looked at the number of times youth were sent to the principal’s office, did “risky” things, fought, cheated on a test or used tobacco. There were no significant overall impacts on these outcomes. (See Appendix B.) Weekly Hours of Homework Net Impact Follow-up Control Mean We were not optimistic that having a Big Brother or Big Sister would improve a Little Brother or Little Sister’s grades during the study period, since other research has shown that grades are fairly stable over time and are generally not affected by non-instructional interventions like BBBS. However, given the importance of school performance to later success and a desire to identify programs that do improve school performance, we collected data on academic performance by asking the study sample youth what types of grades they typically received, ranging from mostly Ds and Fs to mostly As.19, 20 Weekly Hours Spent Reading Net Impact Follow-up Control Mean School Value Scale Net Impact Follow-up Control Mean Overall .27 4.80 .01 2.46 .69 55.27 Gender Male Female .41 .04 4.73 4.91 .12 -.18 2.05 3.12 1.02* .14 54.29 56.89 .66 -.28 .15 .48 4.54 4.74 4.98 5.25 -.51 -.20 .94* -.26 2.21 2.22 1.86 4.68 .85 -.56 1.27 1.27 55.22 57.74 53.05 55.48 Race/Gender Minority Male Minority Female White Male White Female 23 24 Making a Difference: An Impact Study of Big Brothers Big Sisters At the conclusion of the study period, Little Brothers and Little Sisters reported 3 percent better grades than did control youth. Little Brothers and Little Sisters reported, on average, a grade point average (GPA) of 2.71, while controls reported a GPA of 2.63. The grades of Little Sisters, especially minority Little Sisters, appeared to be the most responsive to participation in the program. The average GPA for girls in the control group was 2.67; for Little Sisters it was 2.83. The difference was even greater for minority Little Sisters, who had an average GPA of 2.83 compared with 2.62 for minority girl controls. Thus, we can infer that being involved with BBBS begins to improve the youth’s school performance. We also found that BBBS improved the youth’s school attendance. Little Brothers and Little Sisters were significantly less likely to skip classes or a day of school. At the end of the study period, Little Brothers and Little Sisters had skipped 52 percent fewer days and 37 percent fewer classes. As with the other academic outcomes, the impact was larger for girls. On average, Little Sisters skipped 84 percent fewer days of school than did control girls. Minority Little Sisters skipped 78 percent fewer days than their control counterparts, and white Little Sisters skipped 90 percent fewer days than their control counterparts. Results were similar for skipping classes. Research also shows that youth who feel more competent in school tend to be more engaged and perform better. Therefore, we examined changes in Harter’s scale of perceived scholastic competence (1985) to determine whether participating in the program increased a student’s expectations for school success. Table 12 Net Impact of Participation in BBBS on Family Relationships Outcomes Summary Parental Relationship Measure Net Impact Follow-up Control Mean Trust Net Impact Communication Follow-up Control Mean Net Impact Follow-up Control Mean Overall 1.5** 70.65 .64** 23.79 .53 27.76 Gender Male Female 1.83* .99 71.53 69.21 .67** .60 24.22 23.08 .67 .30 28.08 27.23 Race/Gender Minority Male Minority Female White Male White Female .43 .63 3...
Purchase answer to see full attachment