|
Home
Site Map
Privacy Policy Français |
| ¦ Social Research > HBHC Evaluation |
|
• Subject Areas
|
Evaluation of Ontario's Healthy Babies, Healthy Children Program (HBHC) |
|
The first project, completed in 2002, conducted the Short Term Evaluation of the HBHC program. This major study carried out an extensive process evaluation in 37 health units and outcome evaluation of two cohorts of families. It also completed validation studies on two of the assessment instruments used by the program and investigated the impact of HBHC on the integration of services for families with young children in communities across Ontario. The evaluation team is now conducting the Follow-on Evaluation of the program. This study has re-established contact with families from the second cohort of the short term evaluation. This study will maintain contact with these families and assess outcomes for the family and the child when the child is in kindergarten (2005). It is also extending the work on service integration and carrying out additional instrument validation work. The following sections provide an Overview of Ontario’s HBHC Program, a summary of the key findings and a description of the methods used in the Short Term Evaluation and an overview and status report on the Follow-on Evaluation. OVERVIEW OF ONTARIO’S HBHC PROGRAM The Healthy Babies, Healthy Children program screens all children born in Ontario (about 130,000 births annually) using a variety of methods: a prenatal screen, a screen that is completed in the obstetrical ward, and postpartum screen after the mother has been discharged from the hospital. Each new mother receives a call from a public health nurse within 48 hours of discharge from the hospital. Many of these mothers are subsequently visited by a public health nurse. All families receive educational materials and, where appropriate, are referred to other health and social services in the community. Where the public health nurse judges that the family may benefit from additional services, the nurse completes an in-depth family assessment. On the basis of the assessment, the nurse may offer the family home visiting (typically involving visits from a public health nurse and a family home visitor) and referrals to other services. Where other agencies are also providing services to the family, the program makes provision for coordination of services. All activities of the program are voluntary. At any time, the family has the option of refusing services that are offered and/or withdrawing from services. SHORT TERM EVALUATION The following offers some background information on the first Evaluation of Ontario's Healthy Babies, Healthy Children Program. For ease of reference, the information provided is organized under these headings. FOLLOW-ON EVALUATION Recognizing the need to maintain contact with HBHC families and conduct a second outcome evaluation when the children were in kindergarten, the ministry asked The Social and Policy Research division of TNS Canadian Facts and the Centre for Families, Work, and Well-being at the University of Guelph to undertake a follow-on study of HBHC. The follow-on study focuses on the short-term evaluation's second cohort of children born from July 1, 2000 to June 30, 2001 who agreed that we could contact them should another study be undertaken, 3,110 families. The study builds upon the work for the short-term evaluation, but is more limited in scope. The overall objectives of the follow-on evaluation are to:
This study is ongoing. Findings will be reported in 2006.
THE EVALUATION TEAM Ontario's Ministry of Health and Long-Term Care funded the evaluation through the Integrated Services for Children Division. Brenda Ross was the evaluation coordinator. Ron Robinson, President of the The Social and Policy Research division of TNS Canadian Facts, and Dr. Bruce Ryan, Professor in the Department of Family Relations and Applied Nutrition at the University of Guelph were the Co-Principal investigators. The members of the project team and their principal roles were:
Dr. Bruce Ryan and Dr. Donna Lero are affiliated with the University's Centre for Families, Work and Well-Being (www.uoguelph.ca/cfww). PURPOSE AND DESIGN OF THE EVALUATION HBHC Goals and Objectives The goals of the HBHC program are to:
The objectives of the HBHC program are to:
Scope of the Evaluation The evaluation provided the information and structure needed to monitor delivery of services under the program, validated screening and other assessment tools, described short-term outcomes, and built the capacity for the longer-term outcome evaluation:
Structure of the Outcome Evaluation The outcome evaluation involved a comparison of Ontario families with families in other provinces where no program similar to HBHC existed. The data for the comparison group came from the National Longitudinal Survey of Children and Youth (NLSCY). For some outcome measures that were not adequately covered in the NLSCY, the analysis relied upon comparison with established population norms. In addition, other methods were used to see if families in Ontario who received HBHC services were different from those who did not. This combination of methods led to a reliable assessment of the effects of the program. Ethical Review, Informed Consent and Protection of Privacy Description The evaluation plan was reviewed and approved by the University of Guelph and three health units, Ottawa, Peterborough and Toronto to ensure the evaluation met accepted ethical standards. While the review was at times quite detailed, we acknowledge its thoroughness and the constructive attitude displayed by the reviewers. The design was enhanced and the description of the design was clarified as a result of the questions and comments raised by the reviews. The Municipal Freedom of Information Protection and Privacy Act (MFIPPA) required that we sign agreements with each of the health units to further ensure the privacy of the families. COMPONENTS OF THE RESEARCH On-Site Research—Organization of Work and the Environmental Scan Description A senior member of the qualitative team oversaw the work of the team leaders. To ensure consistency across the 37 communities, a common procedure was used to guide the data collection. The procedure included a template to assist team leaders in preparing the reports for each site. An environmental scan of HBHC in each community provided a description of program implementation and service delivery. The results were recorded in a series of summary sheets:
Key Informant Survey
Description The evaluation plan called for two internet surveys with key informants in the 4th quarter 2000 and again in the 4th quarter 2001. Respondents for these surveys included community partners, service providers, advocacy organizations, lay home visitors, public health nurses, and other relevant community stakeholders. The survey focused on program operations. It asked key informants about their perceptions of the various parts of HBHC including screening and assessment, linkages and the referrals, and contribution of public health nurses and lay home visiting. As well, the survey explored how HBHC may have contributed to the integration of services for families within each community. Local researchers identified survey respondents within their communities and provided e-mail addresses, fax or telephone numbers. Potential respondents were contacted by e-mail or, if no e-mail address was available, by fax. The initial contact provided some background on the survey and instructions on how to complete the questionnaire. Each key informant used a unique password to protect the confidentiality of responses. In-Depth Studies To deepen our understanding of the implementation and operation of HBHC, the evaluation conducted in-depth studies of the program in twelve health units. The work focused closely on what helped and/or hindered the delivery of HBHC direct services and on local service integration. The in-depth investigations used focus groups and individual interviews with service providers, public health nurses, HBHC managers and other partners/service providers, family visitors, network committee members and families. The findings from the in-depth studies were summarized in a separate report designed to inform planning in all health unit jurisdictions. Family Survey Description
A specially prepared ISCIS program drew a sample of families in each health unit by randomly selecting families grouped according to their level of risk. The risk levels were calculated separately for each health unit, based upon the Parkyn Postpartum Screen and the results of the Family Assessment Instrument where an assessment had been completed. Staff at the health unit contacted each family in the sample. The families provided written consent to participate in the survey and to release the data on their participation in the HBHC program from ISCIS database. In 2001, a second survey was completed by another slightly larger group of families. This survey used a modified sample algorithm and consent procedure. ISCIS Extracts Description Each health unit maintains an ISCIS database containing information on each child born to families living within its boundaries. ISCIS records family circumstances, screening and assessment details, and data on any referrals or services provided by HBHC. The Ministry's ISCIS programming team provided three programs to extract data from the database:
Instrument Validation Overview The project team conducted an examination of the Parkyn Postpartum Screening Tool, the HBHC Family Assessment Instrument (FAI) and the Nipissing Developmental Screen. Because the validation studies involved a variety of validity types, the team used an integrated model of construct validity initially proposed by Jane Loevinger, and then extended by Samuel Messick, to coordinate its data collection and analysis. Construct validity is seen as an over-arching concept that encompasses and organizes the various forms of validity commonly discussed in texts on testing and measurement. The model suggests that construct validity consists of three basic components: the substantive component, the structural component, and the external component. Figure 1 outlines the relationship between the three components and indicates how each component figures in the validation of the three instruments.
Related to the question of test validity is the issue of test reliability. At the same time as the validity of these instruments was being assessed, we examined some forms of reliability as well. Figure 2 describes the different forms of test reliability and shows how they were assessed in the validation study.
Parkyn Postpartum Screening Tool This study examined the structural validity or integrity of the instrument by looking at the relationships among the Parkyn items and the total Parkyn score. This same analysis also yielded information on the Parkyn’s internal consistency (reliability).
The study examined the external (predictive) validity (sometimes referred to as criterion validity) by assessing the extent to which the Parkyn and the FAI agree. This work assumed the FAI scores were accurate and examined the extent to which the Parkyn, which is much less costly and is completed for all newborns, predicts the FAI score. Of particular interest was the extent to which the Parkyn results in false negatives and false positives. That is, does the Parkyn flag cases that are not deemed at-risk by the FAI, or fail to flag cases that are judged at-risk by the FAI? Family Assessment Instrument The examination of the Family Assessment Instrument also looked at structural validity by examining the results of a factor analysis and the extent to which individual instrument items contributed to the resulting factors and the overall score. Again, the internal consistency (reliability) of the FAI was assessed. The study also investigated the inter-rater agreement (reliability) between two observers using the FAI with the same family. Nipissing Developmental Screen The Nipissing Developmental Screen was designed for parents to use to track their child’s development, and to flag possible problems. This study checked the instrument for understandability by parents, and for consistency in comparison with both professional judgment and the more established Ages and Stages questionnaire. The study focused on the 12-month and 18-month checklists. A sample of families drawn from the ISCIS database identified children who had some regular contact with a person outside the family—a home visitor in the case of high-risk families, or perhaps a day care worker for any family. Both the parent and the other person completed the Nipissing and the analysis examined agreement between the two ratings (inter-rater reliability). A further sample of parents completed the 12-month Nipissing and also the Ages and Stages questionnaire for 12-month old children (external, concurrent validity). The same procedure was repeated when the child was 18 months old in order to examine the long-term stability of the flagging by the Nipissing instrument (reliability), and assess the agreement between family caregivers and external raters who both used the Nipissing instrument (inter-rater reliability). PRODUCTS AND OUTPUT Site Report Description The evaluation prepared an annual report for each health unit, summarizing the findings from four sources:
The site reports analyzed and integrated all the data collected. Each report followed a common format or template and reflected the data unique to each health unit. It provided a context for the health unit by showing the same data for similar health units in Ontario and for all other Ontario health units. The Provincial Summary Description The evaluation team submitted a consolidated provincial report at the end of March 2001. The report summarized the interim findings under three headings:
HEALTHY BABIES, HEALTHY CHILDREN REPORT CARD (Click here for a .pdf version of the Report Card.) A. WHAT IS HEALTHY BABIES, HEALTHY CHILDREN? A child’s early years—from before birth to age 6—are very important. Healthy babies are more likely to develop into healthy children, and healthy children are more likely to grow up to be healthy adolescents and healthy adults. The Healthy Babies, Healthy Children program helps all children in Ontario have a healthy start in life. The Ministry of Health and Long-Term Care launched Healthy Babies, Healthy Children in 1998. The program offers all families with new babies information on parenting and child development and it delivers other services to families who would benefit from extra help and support. Delivered by the province’s 37 public health units, Healthy Babies, Healthy Children provides:
As well, the program encourages more communication among community services, to help make it easier for all families with young children to get the services they want and need. B. DOES HEALTHY BABIES, HEALTHY CHILDREN WORK? In March 2000, the Ministry of Health and Long-Term Care asked researchers from The Social and Policy Research division of TNS Canadian Facts and the University of Guelph to evaluate Healthy Babies, Healthy Children. Is it achieving its goals? Are its services helping children and families? To answer these questions, the research team talked to 6,222 families across Ontario. They interviewed 3,526 public health nurses and lay home visitors who deliver Healthy Babies, Healthy Children services and others who are familiar with the program. They also conducted in-depth studies to find out how the program was working in 12 health units. The research took more than two years. This Report Card on Healthy Babies, Healthy Children summarizes the findings. C. IS HEALTHY BABIES, HEALTHY CHILDREN REACHING ALL FAMILIES IN ONTARIO? Almost. In 2001, 88% of mothers with new babies in Ontario consented to be screened either before or shortly after the baby’s birth. In 15 health unit areas, 100% of families with new babies were screened. In 30 health units areas, more than 90% of families with new babies participated. Across the province, more than 80% of families with new babies also received a phone call from a public health nurse shortly after leaving hospital. All but four health units said they could provide HBHC services in all the languages they need. This means that almost all families in Ontario with very young children have had some contact with Healthy Babies, Healthy Children, regardless of culture or location. D. IS HEALTHY BABIES, HEALTHY CHILDREN SUCCESSFUL AT FINDING THE FAMILIES WHO COULD BENEFIT FROM MORE HELP AND SUPPORT? Yes. When Healthy Babies, Healthy Children first began, the program designers thought they would be able to identify families who would benefit from extra help and support just by screening pregnant women and new mothers. Some families were missed in the initial screening. However, during the phone call to every new mother, public health nurses are able to find most of those families who would benefit from services. Based on experience with Healthy Babies, Healthy Children so far, about 7% of all Ontario families with new babies can benefit from extra help and support. E. ARE FAMILIES WHO WOULD BENEFIT FROM THE HEALTHY BABIES, HEALTHY CHILDREN HOME VISITING SERVICE USING THE SERVICE? Healthy Babies, Healthy Children is supposed to offer home visiting services to those families among this 7% who would benefit most from these services. Home visiting consists of regular visits from a public health nurse and a lay home visitor. How many visits a family receives, and how often, is decided in discussion with the family. On average, families in home visiting receive a 1.2-hour home visit every 18 days. The researchers found that the program appears to have stabilized, with the equivalent of 7% of Ontario families with newborns receiving home visiting. Initially, all sorts of families received home visiting. In 2002, as health units refined their procedures, 59% of the families receiving two or more home visits were the families who staff thought would benefit the most from the visits (compared to 46% in 2001). F. ARE FAMILIES BEING REFERRED TO SERVICES IN THE COMMUNITY? Healthy Babies, Healthy Children refers many Ontario families with young children to a wide range of community services. In 2001, Healthy Babies, Healthy Children staff made 31,479 formal referrals for 14,378 families who would benefit most from these services. One-third of all formal referrals were to breastfeeding, nutrition, prenatal and infant health services; 16% were to parenting programs and services; 15% to medical services, child therapy and development programs; 12% to social, economic and related family supports; and 25% to “other” services. In addition, program staff made 88,704 informal recommendations, connecting families to supports and services informally by letting families know what services are available, or directing them to those services. The services they recommended were: parenting services and programs (32% of all referrals), breastfeeding, nutrition, prenatal and infant health services (24%), medical services and child development programs (8%), social, economic and related family supports (5%), and “other” services (30%). In summary, Healthy Babies, Healthy Children is connecting many families to local services who might not get connected otherwise. G. ARE THE SERVICES AVAILABLE? DO FAMILIES USE THE SERVICES? Recommendations and referrals are important, but are the services available? Do families actually use them? Most of the service providers who were interviewed, thought there are more services for families with children up to age six than there were before Healthy Babies, Healthy Children began and more culturally and linguistically appropriate services for those families. With Healthy Babies, Healthy Children, more families who need extra help and support are being connected to community services and supports. Are there enough services for families? Not always. About 68% of health units said that some families still have problems getting appropriate services. Service providers reported that waiting lists for services have grown and families and service providers may become discouraged by long waits for service. When they put families on long waiting lists for services, most health units (84%) keep the families in the Healthy Babies, Healthy Children program until the other services open up. On the other hand, service providers also reported that some families do not want the services. Lack of transportation, culture and language are also barriers for some families. H. ARE SERVICES FOR FAMILIES WITH YOUNG CHILDREN MORE INTEGRATED? Healthy Babies, Healthy Children is expected to work with other local organizations to increase service integration. Integration means that agencies coordinate their services and plan new services together. By working this way, agencies can minimize overlap, shorten waiting lists and develop whatever new services families need. As a result, when families need services they should not have to track down agencies or tell their story over and over to different agencies. Are services more integrated now? Yes, there has been progress. Agencies in many communities report that, since Healthy Babies, Healthy Children began, there are now fewer gaps and less overlap in services, and fewer misguided referrals. Agencies are communicating more, and coordinating and planning services better than before. In the communities where services are now more integrated, service providers report that the Healthy Babies, Healthy Children program is also more effective. In their view, families and local agencies tend to be more aware of the program, families tend to receive more fitting referrals, and services for families tend to be better. I. WHAT IMPACT HAS HEALTHY BABIES, HEALTHY CHILDREN HAD ON CHILDREN AND FAMILIES? Healthy Babies, Healthy Children is supposed to improve child health, parenting skills and families’ use of services. Is it succeeding? Yes. When researchers compared families who received Healthy Babies, Healthy Children home visiting with similar families who did not receive home visiting, they found better child and family health among home visiting families. Among home visiting families:
J. DO FAMILIES LIKE HEALTHY BABIES, HEALTHY CHILDREN SERVICES? Families with new babies said that they like receiving a phone call from a public health nurse soon after getting home from the hospital. More than 90% found the nurse’s information helpful and 66% found it very helpful. Families who received home visiting said that they liked the service. Almost all said it increased their confidence in their parenting ability and helped them do practical things, such as calming their crying baby or caring for their baby. Families also said that they liked being connected to local services, particularly parent groups. Almost all families were satisfied with Healthy Babies, Healthy Children services and felt they had been treated with sensitivity and respect. Over 90% of home visiting families said that Healthy Babies, Healthy Children staff understood their family’s needs and believed in the family’s abilities and 88% said that they felt they had a reasonable amount of control over the services they received. They also reported that being involved with the program helped them develop knowledge and skills, reduce stress, increase their sense of support and be more a part of the community where they live. The families’ only complaint was that they could not always get some of the local services they wanted, such as child care. K. WHAT DO PUBLIC HEALTH NURSES AND LAY HOME VISITORS THINK ABOUT HEALTHY BABIES, HEALTHY CHILDREN? The public health nurses and lay home visitors who work with families are overwhelmingly positive about Healthy Babies, Healthy Children, and its impact on families. Almost 97% of the people who work with families every day felt that the program helped families be more aware of their strengths and needs, and be more willing to use services. L. OVERALL, IS HEALTHY BABIES, HEALTHY CHILDREN MAKING A DIFFERENCE? Yes. Since Healthy Babies, Healthy Children began:
More families with children up to age 6 are getting local services and supports and more of these families are getting culturally and linguistically suitable services.
It is too early to know the full impact of Healthy Babies, Healthy Children on children’s lives, but the early indications are that more children in Ontario are getting a healthier start in life. To find out more about the longer term results, the Ministry of Health and Long Term Care recently asked the researchers to continue their work. When the children described here reach kindergarten, a follow-up study will look at how well they are doing and whether Healthy Babies, Healthy Children continues to make a difference in their lives. 1In Healthy Babies, Healthy Children, a lay home visitor is defined as someone from the community who is an experienced mother and who has had special training in helping other parents care for their children and use community resources. For further information contact:
TNS Canadian Facts, a TNS Company
© . All Rights Reserved. 900 – 2 Bloor Street East, Toronto, Ontario M4W 3H8 Phone: (800) 268-7115 E-Mail: infocanada@tns-global.com |
Since
February 2000, The
Social and Policy Research division of TNS Canadian Facts
and the University
of Guelph have been conducting an evaluation of Ontario’s HBHC program,
a voluntary early intervention/prevention program for families with young
children.
The
family survey carried out in late 2000 gathered information about outcomes
of HBHC, reactions to contacts and services provided by HBHC, and additional
background on the family. The survey was administered principally by telephone
in English, French and 11 other languages. If the family was unable or
unwilling to complete the survey by telephone, a face-to-face interview
was provided.
