Ways of Knowing - Evaluating outcomes in promoting emotional well-being For a full copy of this research report please contact Helen Tourtouras (9780 2801) or Margo Moore (9780 2796) at Bankstown CHC, Level 4 The Compass Centre Fetherstone St., BANKSTOWN 2200. Fax: 9780 2888 EXECUTIVE SUMMARY: Changing Lives—Keep Your Balance (CL) and Living with Choices (LWC) are two programs that were designed to promote the mental and emotional well-being of women from non-English speaking backgrounds (NESB). These programs have been conducted for the past three and two years respectively and have involved approximately one hundred and forty (140) women from diverse cultural backgrounds. They form part of the Bilingual Community Education (BCE) Program in Women’s Health. The BCE program involves training and employing women from diverse communities to conduct culturally appropriate health promotion programs for women from their communities in their first language. The program aims to promote improved access to health services, improve health literacy and thereby the adoption of healthy behaviours and most importantly promote support networks for the women. The promotion of mental and emotional health amongst NESB women has been identified as a priority in a number of documents. These include Goals and Targets for Australia's Health in the Year 2000 and Beyond (Nutbeam et. al, 1993), the National Women's Health Policy (CDCSH, 1989), and the National Non-English Speaking Background Women's Health Strategy (Alcorso & Schofield, 1991). In its recommendations this latter document indicates the need for more research aimed at identifying effective programs. Further the development of mechanisms for community consultation and consumer participation in program development and evaluation was a high priority emerging from community consultations around the strategy. Little has been done in the area of developing evaluation methods which can be applied to the area of mental health promotion in a multicultural context. The present study aimed to: assess the effectiveness of the bi-lingual community education programs Changing Lives - Keep Your Balance and Living with Choices in achieving their stated objectives and in improving the mental and emotional well-being of women from NESB; assess the effectiveness of the programs across different cultures; examine the usefulness and appropriateness of the evaluation tools used in the programs; identify other support structures that need to be linked in with the programs to improve their effectiveness; develop a model for program development and evaluation through consultations with consumers, bi-lingual community educators, supervisors, coordinators of the programs, and ethnic health staff; and develop evaluation tools that are appropriate for NESB women, for people with low literacy, and for measuring women's mental and emotional well-being. The research methods used in this study included face-to-face interviews with thirty-seven (37) NESB women who had attended one or other of the programs. They came from a variety of ethnic backgrounds including Cambodia, Chile, Greece, Iraq, Lebanon, Macedonia, Palestine, Peru, Poland, Syria, and Uruguay. Focus group discussions and face-to-face interviews were held with seven (7) bi-lingual community educators (BCEs) who had facilitated the programs. Nine (9) key informants were interviewed at length. These were people who are knowledgeable about or experienced in program evaluation, health promotion, or dealing with people from non-English speaking backgrounds. Interviews with past program participants were conducted by bi-lingual researchers in the woman’s first language. The 4-part Interview Questionnaire covered attendance at and expectations of CL/LWC, the women’s views about the programs, their satisfaction with the program and their preceived benefits from attending it, social contacts made as a result of attendance and suggested improvements. Focus group discussions were conducted using a 3-part Focus Group Discussion Questionnaire that elicited general information, an assessment by the BCEs of the impact of CL/LWC discussions about how they made such judgements and questions regarding the effectiveness of the evaluation tools used in CL/LWC. Finally, key informant interviews were conducted using a 3-part Key Informant Interview Questionnaire that included general information, questions regarding experiences in program evaluation, and questions regarding the evaluation of future mental health promotion programs. The preliminary results of the research were presented at an afternoon workshop. This provided an opportunity to obtain critical feedback, to consult with the workshop participants about the recommendations arising out of the research and to exchange ideas about effective and meaningful models for evaluation processes. The research found that by all indications, Changing Lives—Keep Your Balance and Living with Choices were effective in enhancing the mental and emotional well-being of a majority of the participants. CL was able to: raise the women’s awareness of stress and how it affects them; validate the women’s skills and resources in managing stress; improve the women’s skills in enhancing self-esteem, communication, and problem-solving; and help the women make positive changes to promote their social and emotional well-being. LWC was able to: help the women acknowledge and express their feelings constructively; enable women to recognise their rights; and help the women identify ways of taking care of themselves. The study examined the evaluation tools used in the two programs. They include the Faces Sheet, Well-being Scale, BCE Women’s Health Course Evaluation Sheet, BCE Self-evaluation Sheet, BCE Course Evaluation Sheet, and Support Person Evaluation Sheet. Most participants viewed the Faces Sheet and the Well-being Scale as useful, while most BCEs viewed both tools as having very limited usefulness. The discrepancy in the results led us to recommend a re-examination of the reasons for which the evaluation tools were adopted. The other four evaluation instruments were judged as useful by the BCEs. However, we found that these instruments were inconsistently used and that most BCEs prefer oral evaluations over written evaluations. A Six-Query Model of Impact Evaluation was developed in this research. The model consists of a series of six questions that must be addressed by an evaluator while in the process of designing an evaluation of program impact. The 6 questions are: - Who is the evaluation for? - What function (or functions) will the evaluation serve? - What measure(s) or instrument(s) will the evaluation use? - When and how often will the evaluation be done? - Who will administer the evaluation? - What will be done with the evaluation once it is completed? From the key informant interviews, suggestions regarding evaluation tools that are appropriate for NESB women and for people with low literacy were obtained. These suggestions include oral evaluations, pictorial evaluations, evaluation using videos, and evaluation using music. We obtained very limited information with regard to evaluation tools that are appropriate for measuring women’s mental and emotional well-being. The women who attended the programs, the BCEs and the key informants all stressed the appropriateness of oral evaluations. Documentation of observed behavioural changes and self reporting from the participants provides much valuable information on program effectiveness. The different roles and values evaluation processes can have for participants, facilitators and planners were apparent in the responses from study participants. One of the principles underlying the development of programs such as these is the empowerment of women. This principle is reflected in the women’s responses about how they used the information from the evaluation processes. They were not only the object of evaluation but were active subjects. The desire from the BCEs to have more structured feedback about how the evaluations are used also reflects the importance of this principle. Many of the comments about how they had benefited from the program, made by the women at the time they attended the program, were confirmed in the follow-up impact evaluation interviews. Apart from appraising the effectiveness of Changing Lives—Keep Your Balance and Living with Choices in bringing about positive changes to NESB women’s emotional well-being, the present research also shed light on some of the more important issues that are relevant to the evaluative process. From our experience, we found that evaluation is a complex process that needs considerable thought, planning, flexibility, and creativity.   Summary of Recommendations: The study recommends that: 1. Program planners re-examine program and sessional objectives. 2. The two programs be advertised more widely. 3. The two programs be organised more often. 4. Opportunities for physical exercises and practising problem-solving exercises be maximised when designing mental health promotion programs. 5. The BCEs encourage program participants to approach them after each session or after the program finishes should they find a need to talk to the BCEs in private. 6. The importance of choosing venues for conducting the programs that: (a) are centrally located; (b) provide child care facilities; and (c) provide a separate space for the women to ensure privacy be stressed. 7. Program planners re-examine the Faces Sheet and the Well-being Scale in relation to: (a) what function(s) the tools serve; (b) what information the evaluation process needs to generate using these tools; and (c) who will use the information obtained from these tools. 8. The data that will be collected from the BCE Women’s Health Course Evaluation Sheet be collated, made into a report, and presented in a feedback session to all the BCEs concerned by the Sector BCE Coordinator. This feedback session should be conducted once or twice a year. 9. Evaluation instruments that will be administered to program participants be translated into their first language. 10. The BCE Self-evaluation Sheet be revised to include the question "What changes did the women report or did you observe in them?". 11. The Support Person Evaluation Sheet be revised and used as another evaluation tool. The Evaluation Sheet should include the question "What changes did the women report or were observed by the BCE or support person?". 12. The BCEs be provided a support person as often as possible and especially in sessions where potentially distressing topics will be discussed. 13. The support person explicitly inform program participants that they can contact her after the programs have finished should they wish to discuss any aspect of the programs. 14. Coordination between the BCEs and their local Coordinator be improved. The Area BCE Coordinator should be contacted in the absence of the Sector Coordinator to ensure all resources and documents are made available. 15. In the future, face-to-face interviews with a random sample of participants from CL and LWC be conducted two to three months after the programs have finished. We recommend that a person other than the facilitator of the programs conduct the face-to-face interviews. 16. Program planners present, in the program manuals and the BCE training, a clear and thorough discussion of when and how to use the evaluation tools and the importance of setting aside enough time to conduct the evaluation.