We are currently moving our web services and information to Canada.ca.

The Treasury Board of Canada Secretariat website will remain available until this move is complete.

Art and Architecture of Writing Evaluation Reports

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject à to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Introduction to the Course

Rate Yourself

Long Before Writing the Evaluation Report, the Work Begins...

The Role of The Evaluation Report

Exercise: Before You Start

Writing Style: Tone

Component One: Introduction and Context The Program Context The Evaluation Context

Exercise: Program Description

Writing Style

Component Two: Methodology

Exercise: Identifying Limitations

Component Three: Key Findings

Component Four: Key Conclusions

Component Five: Recommendations

Exercise: Developing Recommendations

Seeking Approval

Writing Effective Sentences

Exercise: Writing Effective Sentences

Component Six: Executive Summary

Exercise: Improving the Executive Summary

Component Seven: Action Plan

Final Checklist

Additional Consideration for Report Writing

Resources

Bibliography

Appendix A: Sample Evaluation Report


Introduction to the Course

The course is entitled the Art and Architecture of Writing Evaluation Reports for several reasons. Writing an evaluation report requires some aspects of all the competencies required of evaluators. It requires knowledge of the program being evaluated, evaluation methodology, various methods of analysis, and the demands for accountability or reporting to government. Drawing on all of this knowledge, report writing requires the ability to work as a team and communicate very well. Writing the evaluation report requires an equal amount of sensitivity and toughness. Despite the demands and challenges, it is rewarding to package the story of a program so that it informs decision makers about important matters.

This course is intended to be part of the Centre of Excellence for Evaluation's Learning Strategy for the federal evaluation community. It will also be delivered as part of the development program for new evaluators. It is based on the assumption that the audience has basic knowledge and experience in report writing. The goal of this one-day course is to discuss strategies that will ensure evaluation reports are credible and effective and satisfy the need for information by decision makers. The course is not intended to teach a template for evaluation reports. The teaching is based on the assumption that you have read the case study and the entire workbook prior to the start of the course.

Evaluation is an important part of public sector management. Writing evaluation reports is in some ways the most visible part of the evaluation. The report lives and is used long after the evaluation work is wrapped up. This course focuses on writing a credible report by discussing the importance of evaluation reports, the components of an evaluation report, and the resources for writing an evaluation report. Each element of the course will be discussed and summarized using the exercises provided. Tips for writing and presenting evaluation reports are included throughout the workbook for discussion during the course. You will be alerted to the requirements of an evaluation report with the understanding that you will continue to work on the course material in your departments.

The Treasury Board Secretariat has developed a Guide for Reviewing Evaluation Reports. It is important that you are familiar with these guidelines and requirements.

Learning Objectives

The learning objectives of this course are for you to:

  • understand the role and importance of an evaluation report
  • understand the components of an evaluation report
  • be able to address information needs with timely and well-written reports
  • be able to involve others in report writing and manage the report writing process

Competencies

Writing evaluation reports is a demanding job. It requires you to be both conceptual and concrete in your approach. It clearly calls on most of the competencies identified for evaluators in the report, Competency Profile for Federal Public Service Evaluation Professionals (2002).

Let's look at the following competencies required of evaluators and ask how they pertain to writing the evaluation report:

Organizational Awareness

Evaluation professionals have a solid understanding of their organization and the role played by evaluation. They understand the challenges faced by managers in designing and monitoring the outcomes of programs, policies, and initiatives and know who the key players are for any given project. They comprehend the complexities of internal and external organizational relationships, procedures, and relationships with key stakeholders. Evaluation professionals actively keep abreast of new organizational developments and upcoming initiatives related to evaluation work.

Interpersonal Relations

Evaluation professionals interact with individuals from diverse backgrounds, occupational groups, specialty areas, and programs. They understand the unique contributions offered by others and the importance of developing and maintaining positive working relationships. They approach each interpersonal situation with sensitivity and diplomacy, possessing a genuine respect and concern for others and their situations. Evaluation professionals are skilled at recognizing and defusing potential conflict, using open and honest interactions. They use their facilitative skills in guiding clients, managers, and organizations, working collaboratively and openly.

Communications

An essential part of an evaluation professional's role is communication. Evaluation professionals are dedicated to communicating clearly, transparently, and concisely, translating technical information into comprehensible forms that suit the needs of the audience. They communicate unpopular findings with tact, diplomacy, and clarity, and use sound judgement when communicating sensitive material. They possess the communication skills required to identify and transmit the level of information required to permit timely, accurate decision-making. Evaluation professionals are active listeners and persistently seek a comprehensive understanding of the issue under discussion.

Action Management

Evaluation professionals provide timely findings to management and stakeholders for use in planning and management decision-making. They work efficiently in both independent and group settings, often judging multiple tasks or projects simultaneously. They manage their own time and individual work activities, securing all the resources at their disposal to accomplish multiple objectives in an effective, efficient manner. They set challenging goals and track the progress of their undertakings to ensure that they have the necessary resources to achieve desired results in a timely manner. When their responsibilities include managing teams, they delegate appropriately, guiding and mentoring less experienced colleagues.

Cognitive Capacity / Analytical Capacity

Evaluation professionals plan, design, and implement sound evaluation methodologies to assess and inform an organization's programs, policies and initiatives. They quickly comprehend the objectives of new programs, policies, and initiatives to which they are exposed and the context in which the programs operate. They are adept at systematically collecting and assimilating substantial quantities and types of information. Evaluation professionals use their strong cognitive skills in critically evaluating and interpreting research findings and in identifying gaps in, and limitations of, the evidence. They formulate plausible hypotheses, consider alternatives, and draw appropriate conclusions from research findings.

Teamwork

Evaluation professionals are team oriented, working alongside clients, managers, and stakeholders in facilitating increased effectiveness. They contribute fully to team and client initiatives. They develop and maintain respectful, collaborative, and positive relations with team members, clients, and mangers. Evaluation professionals seek input, share their own expertise, and consult openly, capitalizing on the diversity of experience, knowledge, expertise, and backgrounds of others. They are at ease collaborating with individuals holding different professional viewpoints and work towards achieving consensus when differences arise.

Ethics and Values

Evaluation professionals hold themselves to high ethical and professional standards. They are objective, fair, and balanced when evaluating programs, policies, and initiatives and strive to ensure the information they gather is factual and complete. They thoroughly evaluate the potential for conflict of interest, and continually monitor the objectivity of the evaluation process. They consistently meet their commitments and obligations, and maintain an appropriate professional distance and credibility even in difficult, high-pressure situations. Evaluation professionals treat others fairly, contributing to a climate of trust, acceptance, and respect for others' principles, values and beliefs.

Rate Yourself

Standard

To meet the standards of The Evaluation Policy (April 2001), evaluation reports must present the findings, conclusions, and recommendations in a clear and objective manner.

Guidance

Evaluation reports should be written so that senior managers and external readers can readily focus on and understand the important issues being reported. Please rate your knowledge of and confidence with the following guidelines. Do you consider yourself Fair, Good or Very Good? Please explain your rating.

Guideline Rating of your knowledge and skill to carry this out
Be concise and write clearly
Include only information that is needed for a proper understanding of the findings, conclusions and recommendations
Present the conclusions and recommendations so that they flow logically from evaluation findings
Clearly expose the limits of the evaluation in terms of scope, methods and conclusions
Satisfy, where applicable, Cabinet, Treasury Board submission or external reporting requirements
Provide the reader with appropriate context by describing the objectives and timing of the work, the policy, program or initiative that was evaluated, how it fits into the overall operations of the organization, and its importance
Provide an accurate assessment of the results that have been achieved by the policy, program, or initiative
Contain clear and actionable recommendations, and timing for management action
Provide relevant analysis and explanation of the exposure to risks for any significant problems identified and in respect of key recommendations

Long Before Writing the Evaluation Report, the Work Begins...

An evaluation report requires a balance of the context and need for information. Preparation starts early and there are a number of matters to be considered and decisions to be made before work on the final report begins.

Know the purpose and use of the evaluation report

Understanding the information needs of the decision makers is essential and it is more complex than it first appears. The Art and Architecture challenge involves balancing the predetermined information needs with what, from the perspective of the program, could also be reported. Regardless of the balance between what is being asked, and what you would like to report, the overriding information need is the policy, legislation, or other decisions to be influenced.

There are usually several audiences for an evaluation report. It is a challenge to know each audience and satisfy the information needs of the various audiences. Knowledge of the audiences and their information needs and their importance to writing the evaluation report should be discussed at the outset of the evaluation. The final report should inform all aspects of the evaluation. The evaluation report therefore really begins at the beginning of the evaluation work.

Know the program being evaluated

Writing the evaluation report requires a thorough knowledge of the program being evaluated. To develop valid conclusions it is essential to know the program theory or the assumptions on which a program is based. It is also important to know the logic model or the linkages between the activities and expected outcomes. Findings and conclusions cannot be made on what is good without an understanding of the expectations for results. Knowing what change a program is intended to create, how the implementation of a program has occurred, the social context of a program and how it fits with policy, is equally important to make conclusions.

Know evaluation practice

Evaluation, by definition, is not a template. Appropriate evaluation practice and reporting is determined by a large number of factors. It is important to focus on the main evaluation question and know its breadth and depth. Know whether the evaluation is a study of the implementation or the outcomes, and whether it was requested as a formative (about program improvement) or a summative evaluation (about continuing with a program). Refer to page 21 to review the elements of the evaluation. Each component and the relationship of the components is what defines the evaluation and forms the basis of the evaluation report.

Performance measurement is an important part of evaluation but must not be confused with evaluation. Performance measurement is usually the ongoing activity that allows management to monitor program implementation and present status reports on a frequent basis. It tends to have an operational focus dealing with activities, outputs, external factors or influences that require continual adjustment, and the immediate outcomes that can be directly attributed to the program. Monitoring is also a part of evaluation, but it is not evaluation. It is important to have agreement on terms and use of all terms.

Establish the role of the evaluation report

What is stated in the evaluation's terms of reference about the report? Beware of no direction. You will never be right. Clarify expectations and what can be delivered in a final report before you begin the evaluation work. The nature and scope of the evaluation report should be in the work plan of the evaluation.

The Role of The Evaluation Report

An evaluation report has an important role to play. The written report may be all the information that decision makers have to understand the design, implementation, and results of an expensive government intervention. The report must represent the experience and views of the many perspectives involved with the intervention.

Rutman and Mowbray (1983), among the first to write on program evaluation in Canada, defined evaluation as "the use of scientific methods to measure the implementation and outcomes of programs, for decision-making purposes" (p.12). John Mayne (2003) recently emphasized that evaluation is largely about 'telling the performance story' (p. 13). He states that the performance story needs to convince a skeptical reader that the activities undertaken have made a difference.

The pressure is on to have one small evaluation report do its job well.

Federal documents such as Evaluation Policy (Treasury Board Secretariat, 2001), Guide to the Development of RMAFs (Treasury Board Secretariat, 2001), Reporting Performance to Parliament: Progress Too Slow (Auditor General of Canada, 2000) were reviewed. These documents, which form the basis of this course, continue to stress the role of decision-making. The writing of Carol Weiss (1994 and 1998), which has been very influential on the topic of using evaluation to influence decision-making, will also be brought to this course.Pencil

Exercise:
Before You Start

Ask yourself (and others):


Who is your audience? What decisions are you influencing?













Do you know what information will influence decisions? What are other uses that may be made of the evaluation or the evaluation report?












Writing Style: Tone

Writing an evaluation report that will be read by a variety of people requires a degree of formality. Consider how a speaker's tone of voice conveys information: how "I didn't know you felt like that" could be interpreted in a sarcastic tone, in an affectionate tone, or in a matter-of-fact tone. To be convincing, the writer of an evaluation report should adopt an impersonal and objective stance. The following are factors that affect the tone of your written report.

Positive Versus Negative

Your readers want to know the facts of the program you are evaluating. They want to know what the situation is, not what it is not. Use words positively.

Negative

Positive

The delivery did not arrive on time. The delivery arrived that afternoon.
The volunteer response was disappointing. Forty volunteers took part in the survey.
He can't see you until four. He can see you at four.

Active Versus Passive Voice

Writing in the active voice--the subject is doing the acting--creates a direct and dynamic prose.

Passive

Active

The plan was approved by our clients Our clients approved the plan.

Third Person Writing

The frequent use of the first person pronoun (I, me, my, we, us and our) should be minimized to avoid sounding self-centered. I injects a personal note, which might weaken a report by making it seem merely a statement of one person's opinion and beliefs ("Business Report Writing"). You refers to the person(s) addressed. Since your report may be read by a number of people and by people you do not know well, the indefinite use of you should be minimized. Although there are times when the use of I and you is appropriate to avoid awkward phrasing, a report that is a factual account or summation of information should be written in the third person most of the time.

Informal

Formal

You can tell from the data that... The data indicates that...
We believe that the new policy will... The new policy will...

Component One: Introduction and Context

The Program Context

A program description is essential to provide the context on which the reader may judge the appropriateness of the evaluation and conclusions. An evaluation report must include a description of the program, policy, or initiative being evaluated. It serves to set the scene for the role of the intervention.

A description will include (but is not limited to):

  • a statement of the problem or situation being addressed
  • the program theory or belief about the intervention
  • a logic model or results chain describing the connection between activities and results
  • the intended reach of the intervention
  • environmental factors and risks association with implementation
  • program resources
  • governance of the program

The program description should 'pack a punch'. It captures a large story in a brief presentation.

The description will include controversial issues such as whether there is evidence the intervention will create a change, whether this is an area of high need with very few resources, and what evidence indicated this was a reasonable way for the government to spend public funds.

The Evaluation Context

The evaluation, simply stated, is intended to report on whether public funds have been spent wisely. The evaluation report must therefore address how conducting the evaluation answered the question: were public funds spent wisely? The following are notes on the elements of the description of evaluation.

  • Clearly state the purpose of the evaluation. Depending on the purpose of the evaluation, the focus of the report will be on whether the design of the intervention was a good design, whether it was well implemented, or whether the desired results were achieved.
  • Similar to the purpose, a description of the evaluation will include how results will be used. This may be a legal requirement. It adds clarity to the purpose of the evaluation.
  • Clearly state what the evaluation will accomplish (the objectives). This is sometimes made clearer by stating what it is not doing.
  • Describe the scope (breadth and depth) of the evaluation. Evaluations do not tend to bring upfront the amount of money assigned to the evaluation. The language for a ten thousand dollar effort can sound very similar to the language for a fifty thousand dollar evaluation. It is necessary therefore to use descriptions of the scope such as the reach (number of stakeholders contacted), the time assigned to carry out the evaluation, the issues to be included in the evaluation and also issues not included.
Pencil

Exercise:
Program Description

What do you need to know about the program for the description to be sufficient?

Does the report indicate why the evaluation was carried out?

Writing Style

Bias-Free Language

The tone of your writing should be free of any suggestion of gender, race, religion, age, disability or ethnic bias. For words such as airline hostess substitute flight attendant, for working mother substitute working parent. If you use personal pronouns referring to a man as a boss and a woman as a nurse, you are stereotyping. The following examples demonstrate problems and suggestions for correction:

Show Bias

Improved

The manager used his initiative. The managers used their initiative; or

The manager showed initiative.

The nurse is required to keep her records up to date. Nurses are required to keep their records up to date; or

The nurse is responsible for updating records.


Awkward

Improved

He or she is to report to the section manager. Telemarketers report to the section manager.

Superlatives also show bias. Try to avoid phrases such as "We are extremely pleased" and "The unit was beautifully written." Your argument, in general, should convey information, not your feelings or slant. When editing your report, many words showing emotion can be excised.

Show Bias

Bias Free

The collectors achieved an outstanding response rate of 50 percent. The collectors achieved a response rate of 50 percent.

Exercise

Improve the following sentences.

  1. A journalist is stimulated by his deadline.
  2. In my opinion, funding needs to be doubled.
  3. If you do not double the funding, the staff must be cut.
  4. It said in the newsletter that the union will vote on Tuesday.
  5. We did not survey the secretarial staff.
  6. Editing is done by the assistant manager.
  7. The enormous increase of 32 percent proves the publicity department has done a superb job.

Note: Remember that #5 may be correct depending on the context. The writer may have elected to use we to avoid awkward phrasing, and it may be necessary to report what the evaluator did not do.

My Sentences

1.

2.

3.

4.

5.

6.

7.

Component Two: Methodology

Sampling, Design and Data Collection

Simply stated, the method section describes how the evaluation was carried out. It explains in detail the work undertaken to collect all data upon which findings, discussion, and recommendations are based.

The method section of the evaluation report is important because:

  • it allows others to assess the extent to which findings are valid and reliable
  • it allows limitations of the evaluation to be clearly defined
  • it allows the evaluation to be replicated
  • it allows identification of alternative explanations for findings
  • it signifies research integrity

The method section describes the evaluation's design and includes a discussion of:

  • the appropriateness of the chosen design for addressing the evaluation questions
  • how participants were selected
  • how participants were assigned to groups or conditions
  • the treatment, program or intervention received by participants in various conditions
  • how measures of outcomes were identified
  • data collection methods and instruments
  • how data were analyzed
  • decisions made about addressing/reporting conflicting or atypical findings
  • limitations of the methods used (sample size, lack of pre-test data, etc)

The method section should outline all aspects of the evaluation such that the reader is able to assess the extent to which observed outcomes can be attributed to the program. A clear description of method allows the reader to assess rigor of design, data quality, and subsequently validity and reliability of conclusions drawn from findings. The method should be described with enough detail for the evaluation to be replicated. The following diagram illustrates the elements of the evaluation that should be covered by the method section. The figure also depicts the elements of the evaluation report.Figure 1: Elements of the Evaluation and Components of the Evaluation Report

Limitations

One challenge of writing the method section is describing the limitations of the evaluation. It can be difficult to provide an objective critique of limitations when you are closely involved with an evaluation study. However, Dooley (2001) notes that integrity implies a need to be completely honest about methods, which means reporting not only on the advantages of the methods employed but also on the disadvantages.

It can even be advantageous to address design limitations upfront in an evaluation report. Discussing what evidence the evaluation provides as well as what evidence it does not provide can help guard against misuse of the findings.

It is important when discussing limitations to identify actual and potential biases of the data. Furthermore, those biases should be explained in terms of their influence on evaluation findings.

Pencil

Exercise:
Identifying Limitations

What are the critical elements to describe in your methodology section?

Identify any limitations of the evaluation design in the case study. Discuss how you would address them in the report.

Component Three: Key Findings

The key findings section of the evaluation report provides the reader with a summary of the evidence upon which your conclusions and recommendations are based. The length and detail of the key findings section will vary depending on the nature of the data collected for the evaluation. For example, while quantitative data can generally be summarized succinctly using traditional statistical reporting methods, reporting on qualitative data in a manner that is concise and thorough can be a challenge. However, traditional techniques for qualitative analysis (e.g. thematic analysis) can be useful for reducing large amounts of data to a manageable size.

Evaluators are sometimes asked to quantify qualitative findings, for example, to note how many interview participants expressed a particular opinion. Qualitative data is generally intended to provide context for the interpretation of other findings or provide an account of the experience of the program. In most cases to quantify qualitative findings is a misuse of qualitative data as doing so is more likely to discredit important findings than contribute meaningful information to the evaluation. For example, suppose the sample evaluation report included a write up of findings of interviews with staff of the Solar Academy (six instructors and two supervisors - principal and vice-principal). If it was reported that "two respondents felt instructors were under-qualified while 75% of respondents felt instructors were adequately qualified", it becomes easy to disregard the opinion of the two respondents. However, if the findings state "while instructors felt qualified for their work, those in leadership positions expressed concern with regard to instructor's qualifications" the opinions of the respondents take on a different meaning.

Readers expect that the evaluation findings will provide answers/insights into the evaluation questions and issues. When deciding what findings are key and therefore require reporting, it is helpful to return to the evaluation questions or issues defined at the onset of the evaluation for guidance.

Key findings should generally be reported concisely and without discussion. The findings should be clear and use plain language as much as possible. They should be presented in a manner that is objective and allows the readers to derive their own conclusions about their implications for the evaluation issues. All findings reported should be testable using methods described in the method section of the report. For this reason, it is necessary to maintain a balanced approach to reporting details.

Component Four: Key Conclusions

The key conclusions section of the evaluation report provides a venue for discussion of plausible interpretations of the findings of the evaluation. Therefore, it is imperative that conclusions drawn by the evaluator are clearly validated by the evaluation findings. This section of the report should provide the reader with meaning and context for the findings presented.

One of the greatest challenges in generating conclusions from evaluation findings is being conscious of making assertions about findings without acknowledging other plausible interpretations. Causal inferences should not be made unless all other plausible explanations have been explored and rejected based on empirical research and/or theory.

Component Five: Recommendations

There is not a 'right' way to present recommendations. How recommendations are to be written requires discussion and agreement. Discussion may include whether recommendations are very specific or whether specifics are a management prerogative. The writing of recommendations must be set in departmental or program demands.

The following information was taken from the Guide for the Review of Evaluation Reports (Treasury Board of Canada Secretariat, 2004).

Recommendations must be based on evidence as follows:

  • They are supported by and flow logically from the report's findings and conclusions.
  • They address significant issues, i.e. they are not 'shopping lists.'
  • To the extent possible, an assessment of the potential impact (on the policy, program or initiative evaluated) of implementing each recommendation is provided.
  • Recommendations incorporate future opportunities, areas for improvement, and future funding or resource possibilities.

Other considerations regarding recommendations include:

  • The recommendations include proposed timing for management action and some indication of quantity and quality.
  • The recommendations are practical and can be implemented.
  • The recommendations are addressed to specific parties.
Pencil

Exercise:
Developing Recommendations

Develop three recommendations based on the key findings and conclusions. Why do you think these are appropriate recommendations?

Seeking Approval

Part of the collaborative process of writing evaluation reports is to seek input on a draft edition. This is delicate because you must be prepared for reaction with which you may not be in agreement. On the other hand, you may find yourself in a very productive discussion. Asking for input on a draft is a good method to get buy-in. When submitting your draft report, it is suggested that you include a table similar to the example below. This offers those reviewing your report an opportunity to note specific concerns, and an opportunity to provide suggestions for changes.

Page # / Paragraph # Specific Comment Issue/Problem with Comment Suggestion for Change

















































Writing Effective Sentences

All reports must be written in good English, using short sentences and correct grammar. Good style is more than good grammar, however. Facts need to be presented in the best possible way; the writer should choose the best and most economical words and phrasing to keep the reader's attention. A report that is unnecessarily long and confusing risks being judged incoherent and not worth reading.

Parallelism

If two or more ideas are parallel, they should be expressed in parallel grammatical form. Single words should be balanced with words, phrases with phrases, and clauses with clauses.

Balance parallel ideas by linking them with correlative conjunctions (either . . . or, not only . . .but also, both . . . and). Headings must be parallel throughout the document, and subheadings must be parallel to other subheadings in the same section.

Not Parallel

Parallel

Good reports are factual, logical, and demonstrate clarity. Good reports are factual, logical and clear.
My boss advised me either to change my flight or take the train. My boss advised me to either change my flight or take the train.
It is easier to speak in abstractions than grounding one's thoughts in reality. It is easier to speak in abstractions than to ground one's thoughts in reality; or

Speaking in abstractions is easier than grounding one's thoughts in reality.

Problems with modifiers

Modifiers, whether they are single words, phrases, or clauses, should point clearly to the words they modify. As a rule, related words should be kept together.

Misplaced

Revised

You will only need to interview each candidate once. You will need to interview each candidate only once.
The King returned to the clinic where he underwent heart surgery in 1982 in a limousine sent by the Prime Minister's office. (Did the surgery take place in the limousine?) Traveling in a limousine sent by the Prime Minister's office, the King returned to the clinic where he underwent heart surgery in 1982.
Opening the window to let out a huge bumblebee, the car accidentally swerved into an oncoming car. (Did the car open the window?) When the driver opened the window to let out a huge bumblebee, the car accidentally swerved into an oncoming car.
One of our executives has been sent to a counselor with a drinking problem. One of our executives with a drinking problem has been sent to a counselor.

Variety of sentence structure

Most sentences in English begin with the subject and are followed by the verb. Putting too many in a row can become monotonous. Try beginning some sentences with adverbial modifiers (words, phrases, or clauses). Also vary the length of your sentences. Too many short sentences may make the report sound like a ninth-grader's. Long sentences with too many details may be confusing. Try to achieve a balance between long and short. For example:

Adjectives start the sentence: Dejected and withdrawn, Edward nearly gave up his search for a job.

A clause starts the sentence: Because Monday was a civic holiday, the committee postponed the meeting to Tuesday.

Wordiness

Ben Franklin once wrote, "Never use a longer word when a shorter word will do." Sir Winston Churchill wrote in August 1940, "To do our work, we all have to read a mass of papers. Nearly all of them are far too long. This wastes time, while energy has to be spent in looking for the essential points. I ask my colleagues and their staff to see to it that their reports are shorter." ("The plain English guide to writing reports")

Expletives are words used to get a sentence started but are vague and meaningless. Expressions such as here are, it is, it is noted that should be excised from your writing.

For example:

Wordy

Better

There are reasons for the popularity of the program. The program is popular because it has been well advertised.
It was published in the local newspaper. The local newspaper published the manager's article.
There is another module that tells the story of Charles Darwin and introduces the theory of evolution. Another module tells the story of Charles Darwin and introduces the theory of evolution.


Redundancies such as close proximity, basic essentials, and true fact are a common cause of wordiness. Phrases such as in my opinion or one must admit that have little or no meaning. Inflated phrases such as in the final analysis can be reduced to finally.

Wordy

Better

The best teachers help each student to become a better student both academically and emotionally. The best teachers help each student to grow both academically and emotionally.
We will file the appropriate papers in the event that we are unable to meet the deadline. We will file the appropriate papers if we are unable to meet the deadline.

Change jargon, which is language used among members of a trade, profession, or group, to plain English.

Jargon

Revised

All employees functioning in the capacity of work-study students are required to give evidence of current enrolment. Work-study students must prove that they are correctly enrolled.
Pencil

Exercise:
Writing Effective Sentences

Improve the following sentences:

Example

My Sentence

The project's fixed costs include material, salaries, advertising, bonus packages for anyone who goes above and beyond the call of duty, and the cost of travel to different cities.
Use the telephone to answer customer questions, email to send order confirmations, and take orders using our Web page.
At the age of eight, I began teaching my children about business.
Customers who complain often alert us to changes we need to make.
When the Web page is finalized it is recommended that it be routed to all managers for final approval.
It is necessary that we reach a decision about whether or not it is desirable to make a request that the office be allowed the opportunity and option of hiring additional workers.
A decision was reached by the committee.
Calling ahead of time, the reservations can be made efficiently.
The president very hurriedly scribbled his name, address, and phone number on the back of a greasy napkin.
The results in all-important essentials are positive.

Component Six: Executive Summary

The Report in Miniature

An executive summary is a condensed or short version of the whole report. The executive summary "seeks to let the reader in on what the real significance of the report is--what is the reader expected to respond to? The reader is a decision maker who will have the responsibility of deciding on some issue(s) related to the report. The executive summary must be written with this need in mind." ("Writing the Executive Summary")

The executive summary appears at the beginning of the report and is important because members of your audience often read only the executive summary because they do not have time to read the entire report. Occasionally the quality of the executive summary determines whether the whole report will be read at all. Because the executive summary may appear alone, it must make sense on its own. The executive summary is usually 1-3 pages long or 10% of the length of the original report.

An executive summary briefly includes the following:

  • a description of the policy, program, or initiative evaluated
  • the reason for the evaluation
  • the client and intended audience of the evaluation
  • a description of the methodology and data sources used in the evaluation as well as the limitations of the evaluation in terms of its scope and methodology
  • the main evaluation findings, conclusions and recommendations

Tips for Writing the Executive Summary

  • If you used an outline to write your report, use that outline in writing the executive summary.
  • Copy text from your report into your summary and then edit, or summarize each section and edit.
  • Edit to eliminate secondary or minor points, but use your judgement as to what is essential.
  • Ensure that only material that is in the whole report is used in the executive summary.
  • Link sections and ideas together.
  • Keeping your readers in mind and the different levels of knowledge they may bring to the report. Define assumptions, acronyms, and technical terms. Avoid jargon.
  • Include all essential elements of the report.
  • Provide only essential support for each recommendation.
  • Present information in the same order and in the same style used in the report.
  • Write the summary only after the long report has been written.
Pencil

Exercise:
Improving the Executive Summary

Review the executive summary of the case study provided in your course materials. Discuss the quality of this executive summary. What changes would you make?

Component Seven: Action Plan

Writing the evaluation report is truly a collaborative effort. The final report ought to be one of 'no surprises'. Based on the input from several perspectives, and validating the perspectives, the report builds gradually. The report should be one that helps both the program and decision makers about the future of the program.

Action plans can take the form of discussion guides for groups to follow-up on an evaluation report.

One approach to ensuring evaluation reports are acted upon is for program managers to consider each recommendation and write an action plan for each recommendation.

Action plans should include a management statement of commitment to take action as well as the time frame for action and who will oversee or be responsible for the action. Action plans are sometimes included in the final evaluation report.

Final Checklist

Outline your strategies for bringing your knowledge and skills on writing an evaluation report to your departments.

Application Strategies
Is there agreement on what constitutes a 'good report' within the federal government? What must be done to establish agreement about writing evaluation reports for particular programs?
What can be done at the outset of an evaluation study that will contribute to writing the evaluation report?
What are some ongoing strategies (throughout the evaluation) that help with writing the evaluation report?
How can you get those conducting the evaluation involved early in writing the evaluation report? How do you get them to think about the final report during the evaluation?
Generally, how will you integrate the material from this course in your department?

Additional Consideration for Report Writing

The following are a number of points to consider when preparing your report. Some may be a review of points covered previously in this workbook. Others are points that have not been previously discussed. Before starting any report, you should review these points as they will help make your writing more efficient. The points for consideration have been divided into report preparation, report writing, privacy considerations and style considerations.

Report Preparation

Determine who is expected to read the report (specialists/non-specialists) and what use the reader is likely to make of the report.

Determine if the report is for internal purposes or public distribution.

Determine the purpose of the report.

Develop an outline for your report to ensure that all necessary information is included.

Determine how you will present your findings - by stakeholder group, by themes that emerged, or in some other way.

Decide on what words you will use (i.e., project or program) and what words require capitalization.

Report Writing

Title your report.

The title should convey the subject of the report to the reader. You can have a main title with a subtitle as a means of making the title seem shorter.

Your report should have a title page that should include the following as a minimum:

  • the title of the report
  • the name of the organization for which it was prepared
  • the name of the person(s) who wrote the report
  • the date the report was released.

Your department may have other items that it expects to be included on the title page.

Length and Flow

If your report is long or complex, include a table of contents. The table of contents should list all main sections of the report, included supplementary material.

Ideas in your report (whether in the introduction, findings, or conclusions) should flow logically and smoothly. Material that might interfere with the flow of ideas should be put in a footnote, endnote or appendix.

Abbreviations and acronyms

Abbreviations or acronyms should be shown for the first time with the complete phrase and the abbreviation or acronym in parentheses. For example, the first time you wanted to use TBS, you would write Treasury Board Secretariat (TBS).

If you use a number of abbreviations or acronyms, list them at the start of the document along with any words that may be unfamiliar to the reader or that are used in a specific way in the report that may be different from general usage.

Miscellaneous

A footnote should not extend over more than half a page. If it is longer than half a page, it should be an endnote or an appendix.

To increase readability, use short, easily read lists.

Use tables, graphs and illustrations, where appropriate, to help readers understand the material.

Illustrations, tables and graphs may be grouped in an appendix. However, the best place for them is in the text, as close as possible to their first mention.

Include notes and supplementary information that you believe will be useful to the reader in appendices.

All appendices must be briefly mentioned in the report. Appendices are numbered, generally with a capital letter, in the order that they are first mentioned.

Ensure quotations correspond exactly to the original and are properly attributed.

Include works consulted, as well as those likely to be of interest to the reader (even if not referenced in the text), in a bibliography.

Privacy Considerations

You cannot guarantee confidentiality. EVERYTHING can be made public. Be aware of this when you advise those from whom you are collecting data about anonymity and confidentiality.

If you interview another public servant, the name and interview results can be made public. If you interview someone outside the government, some information can be kept private.

Have any survey developed reviewed by the person in your department responsible for privacy.

Every department has its own policy with respect to privacy so check with your department when in doubt.

Style Considerations

Headings and Sub-headings

Use headings and subheadings to help prepare the reader for what follows. The size and appearance of a heading should match its importance. The same type of heading should be used consistently throughout a document to indicate subdivisions with the same degree of subordination.

Headings that are of equal importance should have parallel grammatical structures.

Limit the number of headings to three or four.

Spacing and layout

Don't be afraid to use white space in your document - it makes the document more readable. Side margins should be at least 3 cm wide. The top margin should be 2.5 cm deep and the bottom margin should be from 2.5 to 4 cm deep.

Determine the amount of space needed for clear separation of paragraphs, headings, extracts and illustrations. Be consistent in the spacing. You can accomplish this most easily by using styles for your headings and for your text.

Preliminary matter is usually numbered with Roman numerals while the body of the text is usually numbered with Arabic numerals.

Avoid underlining for emphasis. Attention can be focused on a particular point with italics, bolding or other typefaces.

Use a maximum of three fonts on a page.

Keep the main report (not including appendices) to a maximum of 25 pages, if at all possible.

Spelling & Grammar check

And finally....check the spelling and grammar. A document published by Training and Development Canada contains the following sentence: "Effective reports help public servants make wise, informed decisions and save departmental resources and time" ("T014 Writing Effective Reports Course Outline" page 2). Chances are the writer used a spelling checker. The trouble is that spelling checkers identify words that don't match their dictionary. If the word is a real word, the spelling checker can't tell if it's the right word for the context, e.g. their versus there as in "We will review there report when we get their." A grammar checker may tell you that you've used the passive voice, but the checker can't tell you if the passive voice is appropriate in the particular sentence (Locker, 323). There is no substitute for careful and frequent editing. Have a colleague help proofread your report.

Resources

Book and Journal Sources

A good dictionary such as the Oxford Concise Dictionary, a dictionary-style thesaurus such as the Oxford Compact Thesaurus, a style manual such as the MLA Handbook, and a writing text (such as Hult, Hacker, or Lunsford below) are essential tools for writers. Try to get the latest editions of reference books and Canadian editions where possible.

Caron, Daniel, J. (1990). Reporting evaluation results to senior management: A successful canvass. Canadian Journal of Program Evaluation. Vol. 5, No. 2, 69-80.

Cleland, Jane. Business Writing for Results. New York: McGraw-Hill, 2003. "How to create a sense of urgency" (cover) seems somewhat subversive; and the notion of an 'empathy index' (52-53), rather repulsive. This book is about writing short pieces - such as memos and business letters - in a manipulative manner.  Not recommended.

Hacker, Diana. A Canadian Writer's Reference. Toronto: Nelson Canada, 1996. This book has chapters on composing and revising, grammatical sentences, effective sentences, word choice, punctuation, mechanics, documentation, and review of basic grammar. Has tab dividers and an index.

Heritage, Katherine. Report Writing in a Week. London: Hodder & Stoughton, 1997. The information in this book is very basic and often questionable (see 41-43). There's no index (always a bad thing). Rather good section on spelling (58). Not recommended.

Hult, Christine A. and Thomas N. Huckin. The New Century Handbook: Brief Edition. Needham, MA 02494: Allyn & Bacon, 2001. Chapters on writing, research, documentation, document design, special purpose writing, writing correct sentences, common grammar problems, effective sentences and words, punctuation, mechanics and spelling, and ESL issues. Tabbed dividers, an index, and "help boxes" make accessing information easy. Also check website <http://www.abacon.com. >

Kupsh, Joyce. Create High-Impact Business Reports, Chicago: NTC/Contemporary Publishing Group, 1995. This book is rather heavy on desktop publishing and the use of a computer to enhance reports, but it has a very good section on style manuals (60-65) and 'parts of a report' (80 and ff.). Nothing on citing electronic sources. Try to find a more recent edition of this book.

Locker, Kaczmarek and Braun. Business Communication: Building Critical Skills. First Canadian Edition. Toronto: McGraw-Hill Ryerson, 2002. A university textbook, Unit Six, Module 24 is a chapter on writing long reports. This well organized, easy-to-read text has an index, notes on sources, a section on "polishing your prose" at the end of each module, and an answer guide to questions.

Lunsford, Connors and Segal. The Everyday Writer: A Brief Reference. Canadian Edition. N.Y.: St. Martin's Press, 1997. This Canadian edition covers frequently asked questions, composing and revising, sentence style, sentence grammar, words/glossary, punctuation/mechanics, wired style/document design, special kinds of writing (including professional business communications) and ESL problems. Had tab dividers and an index.

Seely, John. Writing Reports. New York: Oxford University Press, 2002. This book would make a good stocking stuffer, something to read on the plane between Halifax and Ottawa. It has a table of contents and an index. It is easy to read and offers sound arguments. There are good summaries of each chapter and nice cartoons. Worth $12.95

Shrunk and White. The Elements of Style. 4th Ed. Boston: Allyn & Bacon, 2000. Short, popular, easy-to-read and access guide for writers.

Truss, Lynne.  Eats, Shoots & Leaves. New York: Gotham Books, 2003. "This is a book for people who love punctuation and get upset when it is mishandled" (front jacket cover). Read it for entertainment. Has information on the origins of punctuation, comparisons between British and American usage, and recent developments and trends. Good sections on the placement of terminal punctuation within closing quotation marks and the use of apostrophes and commas. Truss admits that personal style has something to do with punctuation and quotes writers such as Bernard Shaw and Anton Chekhov.

Electronic Sources

"The plain English guide to writing reports." <http://www.plainenglish.co.uk/reportguide.html. > A six-page guide has good sections on presentation and organization of reports.

"Business Report Writing." <http://planet.tvi.cc.nm.us/ba122/Reports/Report%20Writing.htm> An excellent 13-page guide.

"Writing the Executive Summary." <http://www.columbia.edu/~ftg1/WRITING%20EXECUT.SUMMARY.html> A 3-page guideline.

Bibliography

Auditor General of Canada (2000). Reporting Performance to Parliament: Progress Too Slow. Report to the House of Commons, Chapter 19. December. Ottawa

Dooley. David (2001). Social Research Methods. Prentice-Hall, Inc.: New Jersey.

Mayne, J. (2001). Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly. Canadian Journal of Program Evaluation. Vol 16, 1,
(p. 1-24).

Mayne, J. (2003). Reporting on Outcomes: Setting Performance Expectations and Telling Performance Stories.

Rutman and Mowbray (1984), Evaluation Research Methods: A Basic Guide. Sage Publications: CA.

Treasury Board of Canada Secretariat (2002). Competency Profile for Federal Evaluators. http//www.tbs-sct.gc.ca/cee/stud_etud/capa-pote00-eng.asp.

Treasury Board of Canada Secretariat (2001). Evaluation Policy. www.tbs-sct.gc.ca/pubs_pol/dcgpubs/TBM_161/ep-pe_html

Treasury Board of Canada Secretariat (2004). Guide for the Review of Evaluation Reports.

Treasury Board of Canada Secretariat (2001). Guide to the Development of RMAFs

Weiss, Carol (1998). Evaluation, Second Edition. Prentice-Hall, Inc. New Jersey

Appendix A:

Sample Evaluation Report

Executive Summary

The Program

Solar Academy's Weather Analysis and Reporting (WAR) program is a leader among courses designed to provide training to individuals pursuing a career in weather analysis and reporting. Established in 1992 Solar Academy, in addition to providing training opportunities, conducts research on methods for making accurate weather predictions. The WAR program consists of lecture style modules, instruction on a weather prediction tool and laboratory training. The program is delivered over eight months to full time students.

The Evaluation

The evaluation employed a two group, before and after program design. The comparison group was comprised of 2000 and 2001 graduates of the WAR program and the program group was comprised of 2002 and 2003 graduates. A document and literature review was conducted and interviews were held with graduates of the WAR program, employers of graduates of the WAR program, and WAR program instructors. As well, the accuracy of predictions was analyzed.

The evaluation of the Solar Academy WAR program is intended to assess the impact of the CCD laboratory component on the quality of the WAR program, the impact of the new combination of modules on the quality of the WAR program, and the positive and negative aspects of the current implementation methods of the WAR program.

Findings and Conclusions

The program group showed higher performance on the WART and also experienced more favourable employment rates after graduation from the program. The findings, however, reveal that there is no significant difference between the program and comparison groups in terms of confidence for work in the field.

While the program group indicated significantly more access to resources, no differences were found to exist between the program and comparison groups in terms of their ratings of instructor knowledge. Program group students were also found to be more satisfied with the WAR program than comparison group students.

Both graduates and employers of the comparison group suggested that the students could greatly benefit from the hands-on experience afforded by labs or work term placements. Likewise, students and employers of the program group indicate that the graduates have benefited from the addition of the laboratory sessions.

It was noted by both groups that work term placements would be beneficial for students to gain practical experience in the field. Further research would likely be required to assess the feasibility and efficiency of introducing such a component. It was suggested by the program group graduates that the WAR program be extended to a full year program to allow students more time to complete course requirements.

It is important to consider the fact that employers of the program group also reported some challenges related to what they perceived to be over-confidence of program graduates. This was reported despite student profiling documents indicating no change in confidence levels.

An analysis of accuracy of two-day predictions of temperature and precipitation revealed that the program group was significantly more accurate at predicting temperature than the comparison group. However, findings revealed that there was no significant difference between the groups' ability to predict precipitation.

In conclusion, the addition of a laboratory and work term component to the WAR program has generally had a positive effect on the program in terms of student satisfaction and on the performance of graduates from the program. Those graduating from the revised WAR program were noted to experience increased employment opportunities, feel more prepared to work in the program and were more satisfied with the program in general. Further research into the effects of various program components on self-confidence, and the relationship between self-confidence and workplace performance is suggested to further inform future decision-making with regard to Solar Academy's WAR program.

The Program

Solar Academy's Weather Analysis and Reporting (WAR) program is a leader among courses designed to provide training to individuals pursuing a career in weather analysis and reporting. Established in 1992 Solar Academy, in addition to providing training opportunities, conducts research on methods for making accurate weather predictions. The Academy revises its approach to educating students of the WAR program to reflect the latest findings of weather prediction theories, thus better ensuring quality programming. In September 2002, a revised WAR program that included two new components was introduced. Prior to these two components being implemented, the WAR program consisted of 32 lecture-style modules, which were delivered over eight months to full-time students.

The first component added to the program was instruction on the use of a new weather prediction tool called a Climatic Conjecture Device (CCD). The CCD is a high-tech computer-software program that predicts weather using data collected from satellites, a historical weather database, and barometric readings. Previous empirical research has found the program to improve accuracy in weather prediction when used with other traditional weather-prediction instruments (Blusky, 1997).

The second component added to the WAR program is a revised approach to the presentation of weather-prediction findings. Based on social science research findings, Solar Academy has identified a number of competencies and reporting strategies for weather reporters that positively affect audience satisfaction with weather reporting. In an attempt to teach these competencies and strategies, Solar Academy has added the following modules to the WAR program:

  • How to be satisfyingly vague - this module provides tips for weather reporters that allow them to appear to deliver a detailed five-day forecast without setting audience expectations too high. Based on expectancy theory, the techniques learned in this module are intended to contribute to the reporters' skills in balancing between accuracy and detail in a manner that maximizes audience satisfaction.
  • How to increase perceived accuracy - a follow-up to the previous module, this module is intended to provide weather reporters with the necessary skills for increasing perceived accuracy of the weather reports. The module covers techniques for recapping in an advantageous manner, avoiding questions regarding past instances of inaccuracy and using broad definitions of "good weather".
  • Effective use of weather props - this module uses role-playing and other theatrical learning techniques to perfect participants' use of weather forecasting props. The module gives tips for being entertaining and not cheesy, switching props quickly and without distraction, and reading the prompting monitor while wearing oversized sunglasses.
  • Delivering bad news in a good way - this module provides participants with skills in softening bad news. It focuses on how tone of voice, facial expressions, and hand movements can be used to minimize audience irritation and frustration following the delivery of an unpleasant forecast.

These two program components have been included in the curriculum of two graduating classes so far (classes of '02, and '03). To accommodate the addition of these new components, the Academy removed two existing modules and added a laboratory component to the program which would allow instruction on the use of the CCD. The Solar Academy is evaluating the program in an attempt to assess the impact of the CCD on students' accuracy in weather prediction and the impact of the new components on program quality. A software production company provided the Academy with the CCD software package free of charge on a trial basis. However, the contract between the software company and Solar Academy is up for renewal, and to receive an updated and supported version of the program, Solar Academy must purchase the software package at a price of $3500.00 per unit installed and pay a yearly fee of $150.00 for technical support, updated versions of the program, and virus protection designed to specifically address "weather bugs" intended to destroy this particular software. The Solar Academy is interested in assessing the impact of the CCD on the program and its students, as well as the process by which the CCD has been incorporated into the WAR program.

The revised weather-reporting training modules described above are also being evaluated. The Academy is looking to improve the delivery of all modules of the WAR program and is particularly interested in how the new modules have been incorporated into the existing program.

The evaluation of the Solar Academy WAR program is intended to answer the following general evaluation questions:

  1. What is the impact of the CCD laboratory component on the quality of the WAR program?
  2. What is the impact of the new combination of modules on the quality of the WAR program?
  3. What are the positive and negative aspects of the current implementation methods of the WAR program?

To answer these questions, it is first necessary to identify indicators for assessing the quality of the WAR program. The evaluators identified the following indicators based on literature and document review and discussions with weather-analysis experts and program personnel:

  • Audience satisfaction with weather reports
  • Employer satisfaction with employee performance
  • Score on the Weather Analysis and Reporting Test (WART)
  • Student satisfaction
  • Student confidence
  • Student preparedness

Logic Model for the Solar Academy WAR program



MAIN COMPONENTS


Climatic Conjecture Device (CCD) Laboratory



WAR program modules


Arrow Arrow

IMPLEMENTATION OBJECTIVES

To teach students appropriate use of the CCD for making weather predictions To provide students with the skills necessary for effective weather analysis and reporting

Arrow Arrow

OUTPUTS


  • # of students completing the laboratory assignments
  • # of students displaying knowledge of effective use of the CCD

 


  • # of students displaying skills appropriate for effective weather analysis and reporting
  • # of students who feel adequately prepared to enter the workforce

 


Arrow Arrow

PLANNED SHORT-TERM OUTCOMES


  • Accuracy of weather reports are improved
  • Increased employer satisfaction
  • Increased audience satisfaction
  • Increased student satisfaction with WAR program



Arrow

PLANNED LONG-TERM OUTCOMES


  • Increased employment rate of Solar Academy graduates
  • Improved reputation for Solar Academy
  • Increased enrollment at Solar Academy


Methodology

This section outlines the steps undertaken in the evaluation of the WAR program.

Project Management

The evaluation was conducted internally by an evaluation committee, which worked under the direction and expertise of the Academy's lead evaluation advisor. To ensure accuracy and usefulness of the evaluation report, the evaluation committee collaborated with program staff to:

  • Identify documents for review
  • Identify and contact data sources
  • Discuss and approve interview guides to be used during data collection
  • Discuss and approve draft and final evaluation reports

The Design

The evaluation employed a two-group, before and after program design. Because the Solar Academy has kept detailed records of students' achievements and maintains an up-to-date database of graduates, the evaluation committee was able to compare the experiences of 2000 and 2001 graduates of the WAR program (the comparison group) with those of 2002 and 2003 graduates (the program group). Furthermore, the evaluation assessed employer and audience satisfaction with members of the program and comparison groups.

Data Collection

Literature Review

The evaluation committee reviewed literature related to the development of effective training programs for weather analysts and reporters, as well as literature related directly to the CCD. This review of the literature was used to inform evaluation tool development and the analysis and interpretation of evaluation findings.

Document Review

The evaluation committee reviewed documents regarding the WAR program (i.e., data-bases, yearly program implementation documents, and student evaluation reports) as a means of assessing changes in implementation between 2000 and 2003. Findings of the comparison group student-evaluation reports were compared to those of program group student evaluations to assess changes in student satisfaction and confidence that correlate with the addition of the two new components to the program.

The document review also included an analysis of WAR student scores on the Weather Analysis and Reporting Test (WART) between 2000 and 2003. The WART is a standardized test designed to assess skill level in weather analysis and reporting. This exam must be passed by every individual wishing to become a weather analyst and reporter in North America.

Interviews

Interviews were conducted with graduates of the WAR program and employers of graduates of the WAR program. Each member of the WAR program student database who graduated between 2000 and 2003 was contacted to participate in the evaluation. The evaluators explained the purpose of the evaluation and asked each graduate to participate in the evaluation by participating in an interview regarding their experiences with the program. A total of 100 graduates were contacted (25 graduates each year), and 72 of those contacted agreed to participate in the evaluation. Of those that participated, 34 were comparison group members and 38 were program group members.

Current and prior employers of WAR program graduates were also contacted to discuss their satisfaction with the knowledge and expertise of program graduates. Fifty employers of program graduates were contacted to participate in the evaluation; 19 agreed to participate. Of those that agreed to participate, 10 had hired comparison group graduates, seven had hired program group graduates and two had hired graduates of both groups.

Prediction Accuracy Analysis

Using archived weather broadcasts provided by employers agreeing to participate in the evaluation, and the National Weather Board's documentation of actual weather patterns, the evaluation committee was able to assess accuracy in weather predictions with relation to temperature and precipitation. Weather predictions made two days in advance were compared to documentation of the actual weather conditions two days following the forecast for the first 6 months of employment. It was decided that accuracy of two-day weather forecasts would be analyzed due to the fact that emperical research has indicated audience satisfaction is most impacted by the accuracy of two-day forecasts. Thus, this technique was employed as a means of assessing audience satisfaction.

Data Analysis

The data was analyzed using traditional quantitative and qualitative analytic techniques. Findings of interviews with program participants and employers were analyzed for thematic content by three researchers. The themes identified by all researchers were compared, and any diversity in interpretation of findings was resolved through deliberation among the researchers.

Statistical analyses appropriate for assessing trends in student performance, confidence, and satisfaction with the program were conducted on quantitative data such as students' scores on the WART, WAR program student-evaluation reports, and two-day forecast accuracy.

Key Findings

The following section contains a synthesis of data collected for the evaluation of Solar Academy's WAR program. Key findings of data collection are organized in this section by data collection method and are presented without discussion.

Document Review

A review of records of the WAR program revealed that comparison group graduates received 32 learning modules intended to prepare them pursue a career in weather analysis and reporting. Program group graduates received 34 modules and 11 CCD lab sessions. A profile of students graduating between 2000 and 2003 was developed based on a review of the student database maintained by Solar Academy. An analysis of student profiles revealed no significance between the program group and comparison group in terms of class size, age, GPA, previous years of experience. Significant differences were found to exist between male to female ratios, WART scores and employment rates for the two groups. Table 1 provides a summary of the graduating class for each year and average scores for comparison and program groups.

Table 1 Profile of students by year graduated

Year graduated

2000

2001

00-01 mean

2002

2003

02-03 mean

Total # students

25

25

25

25

25

25

Male:Female

18:7

20:5

19:6

13:12

9:16

11:14

Average age

24

23

24

24

24

24

Average GPA(out of 4)

3.4

3.1

3.3

3.6

3.4

3.5

Average WART score (out of 100)

85%

88%

87%

94%

90%

92%

Average years experience in field

1.2

1.5

1.4

1.5

1.1

1.3

Employment rate*

60%

56%

58%

88%

92%

90%

*Employment rate refers to the percentage of graduating students employed in their field within 6 months of graduation

Each year a student-evaluation report is prepared by the Solar Academy's student union. This report provides a summary of the findings of students' evaluations of various aspects of their programs including specific instructor assessments, module ratings, course-material ratings and overall program and academy ratings. Each program is given an score on the constructs. Scores for each construct are calculated based on student responses to a number of items related to the construct. A high score indicates a favourable student-rating on a construct while unfavourable ratings result in lower scores. The highest score possible is 5; the lowest is 0. These documents were reviewed for factors relevant to the quality of the WAR program. Table 2 summarizes the findings of this review of student evaluation reports. An analysis of the student-evaluation reports between 2000 and 2003 indicate significant differences in student ratings in terms of accessibility to resources, instructors' approach to teaching, students' level of preparedness to work in weather analysis and reporting, and overall satisfaction with the program.

Table 2 Summary of student-evaluation report findings by year

Year

2000

2001

2002

2003

Percentage of students completing survey

96

92

100

96

Average student ratings on the following aspects of the WAR Program:
Relevance of modules

Relevance of laboratories

Accessibility to instructors

Accessibility to resources

Instructor's knowledge of topics

Instructor's approach to teaching

Your preparedness to work in your field

Your confidence in working in your field

Overall satisfaction with the program

Overall satisfaction with the school

4

NA

5

3

4

3

3

3

3

4

4

NA

4

4

5

3

4

3

4

3

5

4

5

5

4

4

5

3

4

4

4

5

5

4

5

5

5

4

5

5

Interviews

Comparison Group Graduates

Thirty-four 2000 and 2001 graduates were interviewed to gather their experiences with the WAR program. Respondents were asked to comment on what they felt were the benefits and challenges of the program and to make suggestions for change.

With respect to the benefits of the WAR program, respondents felt that the instructors were of high quality and knowledgeable in their specialized field. They noted that they could contact instructors at any time and that they felt their instructors provided a great deal of support to them. Respondents indicated that the modules were useful for the most part, although they did not feel that the modules on Goal Visualization, Self Confidence, and Resume Building were of any particular benefit to them in subsequent working contracts. Respondents also felt that the theoretical grounding provided by the program had been very useful.

When asked to comment on challenges, 2000 and 2001 graduates noted that the program provided little opportunity for hands-on experience and application of theory to real-world situations. They also noted that some of the modules were very challenging, although it was generally felt that this was necessary given the nature of the modules. Students also noted a great deal of animosity between themselves and the students of the Solar Panel and Windmill Construction Program. This animosity was noted to be deep rooted and a result of a 10-year history of rivalry between the two programs for possession of a squash trophy that signifies victory in the annual Solar Academy Tournament. Finally, respondents noted that although they were well prepared in terms of weather analysis, they felt there should be more focus on the reporting aspects of the program. When asked what changes they would make to the program, respondents suggested the following:

  • Provide more modules on techniques for reporting on weather
  • Provide more hands-on experience through lab courses or work-term placements
  • Increase opportunities for collaboration with other weather schools
  • Increase access to computers

Program Group Graduates

Thirty-eight 2002 and 2003 graduates were interviewed to gather their experiences with the WAR program. Respondents were asked to comment on what they felt were the benefits and challenges of the program and to make suggestions for change.

With regard to benefits of the WAR program, respondents felt that the laboratory sessions were particularly useful in preparing them for a career in weather analysis and reporting. They noted that the theoretical knowledge they received from in-class lectures was very beneficial, though they felt that without the labs the theory would be less meaningful. Respondents felt that instructors were knowledgeable and dedicated to the program and the students. They felt comfortable asking instructors for guidance and believed instructors were reasonable and just in their assignment of work and expectations of students. Respondents indicated that the modules were relevant and prepared them for dealing with issues faced by employees in this field.

When asked to comment on challenges, program group graduates noted that time was the greatest challenge. Respondents indicated that, as a result of heavy workloads, they did not always feel they could put as much effort into assignments as they would like. Like 2000 and 2001 graduates, they noted that some of the modules were very challenging; again it was felt this was necessary given the nature of the lessons being taught by the modules. Program group students also noted the animosity between themselves and students of the Solar Panel and Windmill Construction Program to be a challenge at times.

When asked what changes they would make to the program, respondents suggested the following:

  • Extend the program over a full year as opposed to an eight-month program
  • Provide work-term placements
  • Provide more social interaction opportunities between programs at the Solar Academy to foster positive relationships between program students

Employers

Employers were asked to comment on the strengths and weaknesses of the graduates of the Solar Academy WAR program. They were also asked to suggest changes to the program that they feel would improve the skills of program graduates.

Employers who had hired comparison group graduates noted that they had a great deal of enthusiasm for their work and that they were dedicated to improving their knowledge through a mentoring relationship with a senior weather specialist. They noted that graduates had an extremely good theoretical background, although the lack of hands-on experience required that they spend a great deal of time training them to use the necessary equipment. However, following the training period the graduates had become very good workers. They noted that the greatest weakness of the employees they had hired from the program tended to be their knowledge of technology and equipment necessary for analyzing weather data. When asked for suggestions, the respondents felt that a work-term component to the program would likely be very beneficial in providing hands-on training for students, thus allowing them to move into better positions in the field immediately following graduation.

Employers who had hired program group graduates noted that these individuals were confident and knowledgeable in their work. It was felt that graduates of the program were well prepared to begin work in analyzing and reporting weather and needed little supervision or direction from senior staff in terms of equipment use and maintenance and reporting etiquette. It was felt that graduates' in-depth knowledge of the CCD software program was one of the greatest strengthens of the graduates. Several employers noted that although they had access to this software for some time now, they had not used it to its full potential as a result of a lack of staff knowledgeable in its application. However, employers also noted that in some cases graduates of the WAR program had behaved in a manner suggesting over-confidence in their abilities. Examples included not heeding the advice of senior weather analysts or making decisions on their own that should have been discussed with management. It was suggested that graduates of the program could benefit from a work term or mentoring component as this might give them experience in the culture of the weather analysis and reporting industry, thus providing them with skills that will foster more appropriate workplace behaviour.

Prediction Accuracy Analysis

An analysis of weather-prediction accuracy revealed that in terms of temperature, predictions made by comparison group graduates were, on average, within 1.4 degrees Celsius of the actual temperature two days later; program group graduates were within 0.5 degrees Celsius of the actual temperature two days later. In terms of precipitation, on average, comparison group graduates made correct predictions about whether there would be precipitation two days later 94% of the time; program group graduates were correct 95% of the time.

Key Conclusions

The evaluation of the WAR program has revealed significant, encouraging changes between 2000 and 2003 in various aspects of the program and the students it produces. For example, an analysis of student-profile data revealed that the program group showed higher performance on the WART and also experienced more favourable employment rates after graduation from the program. While it is impossible to attribute this to the program changes alone, it is likely that program revision was a contributing factor. This conclusion is further supported by evidence from the review of the student-evaluation reports, which indicates that program group students felt better prepared to work in the field of weather analysis and reporting. A review of the literature indicates that one of the factors known to predict course quality is its rating in terms of providing appropriate preparedness to participants (Gess & Rite, 2001). However, confidence in working in the field is also a known predictor of course quality (Ego, 1997). The present findings reveal that there is no significant difference between the program and comparison groups in terms of confidence for work in the field. This can likely be explained by the greater proportion of women in the program group, as research has indicated that women tend to have less confidence than men in work settings despite the quality and level of training they have received (Shai & Humbell, 2000).

Other predictors of high-quality instructional programs identified by Gess and Rite (2001) include access to resources, instructor knowledge, theoretical and practical components, and student satisfaction. While the program group indicated significantly more access to resources, no differences were found to exist between the program and comparison groups in terms of their ratings of instructor knowledge. Program group students were also found to be more satisfied with the WAR program than comparison group students, thus increasing our confidence that the addition of the laboratory component and the new mix of teaching modules has improved program quality.

The addition of the laboratory component adds a more practical instructional approach to the program. Both graduates and employers of the comparison group suggested that the students could greatly benefit from the hands-on experience afforded by labs or work-term placements. Likewise, students and employers of the program group indicate that the graduates have benefited from the addition of the laboratory sessions. This benefit is evident in the students' high ratings on relevance of the lab exercises and employers confirming the ability of students to use equipment unsupervised and apply technology in new ways to make the work of the company more effective and efficient through the use of the CCD.

Although the revised WAR program contains a practical component that was not available to students in the past, it was noted by both groups that work term placements would be beneficial for students to gain practical experience in the field. Further research would likely be required to assess the feasibility and efficiency of introducing such a component. The program group graduates suggested that the WAR program be extended to a full-year program to allow students more time to complete course requirements. If further exploration into the feasibility of a work-term component is carried out, it would be worthwhile to assess how the work term may allow the program to be extended. Shai & Humbell (2000) ascertain that increased confidence is positively correlated with practice and time spent preparing for a task. This is relevant in explaining why confidence levels of the program group were not significantly different than those of the comparison group. Shai and Humbell's findings suggest that one way confidence could be increased is by providing students with more practice and time to prepare for working in their field. This is beneficial to the program because confidence is directly related to job performance in such a way that increased confidence results in increased job performance. Increased job performance will likely promote the reputation of Solar Academy and its graduates among potential employers. However, it is important to consider the fact that employers of the program group also reported some challenges related to what they perceived to be over-confidence of program graduates. This was reported despite student profiling documents indicating no change in confidence levels. The origin of such conflicting information is worth investigating before attempting to address issues related to confidence in the workplace.

An analysis of accuracy of two-day predictions of temperature and precipitation revealed that the program group was significantly more accurate at predicting temperature than the comparison group. However, findings revealed that there was no significant difference between the groups' ability to predict precipitation. A review of the literature indicates that audience satisfaction with weather analysis and reporting is most significantly impacted by accuracy of two-day forecasts (Storm & Vane, 1992). However, further empirical research has indicated that accuracy in terms of predicted precipitation is a much stronger predictor of audience satisfaction than temperature accuracy (Rane, Sonny, & McCloud, 2002). Therefore, the impact of the increased accuracy of program group students' temperature predictions may or may not have an impact on audience satisfaction with weather analysis and reporting.

In conclusion, the addition of a laboratory and work-term component to the WAR program has generally had a positive effect on the program in terms of student satisfaction and on the performance of graduates from the program. Those graduating from the revised WAR program were noted to experience increased employment opportunities, to feel more prepared to work in the program and to be more satisfied with the program in general. Additionally, program group graduates showed superior temperature-prediction accuracy and were viewed as somewhat more competent and prepared to work than those graduates of the comparison group. Further research into the effects of various program components on self-confidence, and the relationship between self-confidence and workplace performance is suggested to further inform future decision-making with regard to Solar Academy's WAR program.


Date modified: