Evaluation of the Harvest Sample Program - Final Report

Executive Summary

Overview of the evaluation

The Canadian Grain Commission was established by the Canada Grain Act in 1912 as the federal government agency mandated to, “in the interests of producers, establish and maintain standards of quality for Canadian grain and regulate grain handling in Canada to, ensure a dependable commodity for domestic and export markets.”

The Harvest Sample Program (HSP) has been administered by the Canadian Grain Commission since 1927. This program provides the Canadian Grain Commission with information on the processing and end-use quality of Canadian grains harvested each year. The Harvest Sample Program runs from March to November each year. Samples from the crop of western and eastern grains are solicited, collected, and analyzed. The Canadian Grain Commission uses these samples to develop visual grading standards, monitor and support the quality assurance system, and support grain research activities. Participating producers receive unofficial grades and quality information for each sample they submit. This information assists producers with marketing their grain. Producers can also compare this information to the grade potential buyers may offer for their grain. Harvest quality reports resulting from analysis of samples are provided free of charge to interested stakeholders such as grain companies, buyers and processors to support the marketing and use of Canadian grains.

The Canadian Grain Commission’s Audit and Evaluation Services unit included the evaluation of the Harvest Sample Program in its approved Evaluation Plan for 2015-2016. The evaluation objectives and methodology were developed in accordance with the Treasury Board Policy, Directives and Standards on Evaluation (2009). The evaluation results inform future program planning.

Audit and Evaluation Services hired the consulting firm Ference & Company of Vancouver, B.C. to conduct the evaluation.

Evaluation methodology

The evaluation examined the relevance, performance, efficiency and economy of the Harvest Sample Program for the period from April 1, 2010 to November 30, 2015. The evaluation was based on:

  • a review of federal government, Canadian Grain Commission and program documents and data
  • a literature review and comparison of similar programs in other jurisdictions
  • interviews with 28 Canadian Grain Commission representatives, 19 industry and producer associations, 21 domestic grain companies and buyers, and 17 international buyers of Canadian grain
  • surveys of 1,207 producers who have participated in the program by sending harvest samples
  • surveys of 125 producers who registered but have never submitted samples to the program

Findings and conclusions

Relevance

The Harvest Sample Program is aligned with federal roles and responsibilities, federal priorities and Canadian Grain Commission strategic outcomes. The Canada Grain Act does not mandate the program, but it supports key legislated responsibilities of the Canadian Grain Commission. These include developing visual grading standards, monitoring and verification of grading factors, research on environmental conditions, and other special research projects.

The Harvest Sample Program is aligned with federal priorities to ensure the growth, competitiveness and sustainability of the agriculture sector. It supports the Canadian Grain Commission’s strategic outcome to ensure that, “Canada’s grain is safe, reliable, and marketable and Canadian grain producers are properly compensated for grain deliveries to licensed grain companies.” The program also supports the Canadian Grain Commission’s organizational priority, “investing in stakeholder relations”, as identified in its 2015-16 Report on Plans and Priorities.

There is a significant need for the Harvest Sample Program to continue. The samples are essential to the Grain Quality Research Program and Quality Assurance Program. The Harvest Sample Program is the primary or only source of materials for many grain quality research projects. The program samples are ideal for research because they are pure, non-blended samples with a known geographic location obtained directly from producers at the beginning of the supply chain. Other sources of materials such as cargo shipment samples, plant breeder samples, and composite samples obtained from elevators, producer associations or other stakeholders would not suit the Canadian Grain Commission’s needs.

The database of registered producers has decreased, mainly due to the retirement, relocation and death of producers. This has resulted in a 24% decline in the number of producers registered for the program in 2015. The survey has found that producers in the program tend to be older than the average overall Canadian farming community. Attracting new producers to the program has been a priority but the program must put more effort into recruiting younger producers.

Changes in grain marketing and the introduction of an open market for western wheat and barley have led to increased demand for the Harvest Sample Program results from organizations that use this data. Use of the free annual harvest quality reports is significant and increasing. The Harvest Sample Program complements harvest surveys conducted by other organizations in Canada. Some of the information in the Canadian Grain Commission’s annual harvest quality reports and the Canadian International Grains Institute’s harvest assessment report overlaps. However, the evaluation found that these reports are used differently by the same stakeholders and benefit different stakeholders in the value chain.

The differences in the harvest sampling methods used by the Canadian Grain Commission and the Canadian International Grains Institute are complementary. The Canadian International Grains Institute partners with 9 major grain companies to obtain samples and provides them with grading information to support their marketing. The Harvest Sample Program obtains samples directly from producers and provides them with unofficial grades to inform their marketing strategies in alignment with the Canadian Grain Commission’s producer protection objectives. The Canadian International Grains Institute’s annual assessment and report includes only Canada Western Red Spring, Canada Western Amber Durum, Canada Western Red Winter and Canada Prairie Spring Red wheat from the Prairie region. In addition to these 4 classes, the Canadian Grain Commission publishes harvest quality reports for other western Canadian wheat classes, canola, flaxseed, lentils, malting barley and peas, Ontario wheat, and Canadian non-food grade soybeans and food-grade soybeans.

The Grain Farmers of Ontario’s Ontario Wheat Harvest Quality Scoop is also complementary to the Harvest Sample Program. It is carried out in collaboration with the Canadian Grain Commission as an extension of the Harvest Sample Program. However, obtaining enough samples from producers in Ontario and Quebec has been a challenge. The Canadian Grain Commission may need to enhance its relationship with Grain Farmers of Ontario and build new relationships to improve the Harvest Sample Program for eastern grains. Other sources of grading and assessment, such as grain companies and analytical labs, do not duplicate or overlap with the Harvest Sample Program. These sources have a different purpose and do not publish harvest quality information. Almost all recipients use the information the Canadian Grain Commission provides to complement other sources of harvest quality information.

Achievement of intended outcomes

The Harvest Sample Program has been successful in achieving its immediate outcomes to:

  • increase producers’ knowledge of their grain quality
  • increase awareness among domestic and international buyers and processors of the quality of Canadian grain crops
  • support the activities and objectives of the quality assurance and grain quality research programs

The Harvest Sample Program provides the quality assurance and grain quality research programs with an annual source of unblended producer samples. This maximizes the range of varieties, environmental factors, and quality characteristics of the samples. The annual harvest quality reports and information generated using the Harvest Sample Program samples also support the Canadian Grain Commission’s obligation to implement a grading system that supports the efficient marketing of grain within and outside of Canada.

External program beneficiaries are very satisfied with the Harvest Sample Program overall. Almost all producers find the unofficial grade and quality information useful or very useful in providing information to better market their grain. The annual harvest quality reports provide very useful information on the Canadian grain crop to external recipients.

The Harvest Sample Program has also been successful in achieving its intermediate outcomes:

  • improve producers’ ability to negotiate a price and grade for their grain
  • increase stakeholders’ confidence in the quality, grading factors and specifications of the Canadian grain crop
  • provide information that assists the domestic grain industry in marketing Canadian grains

The free, unofficial grade and quality information participating producers receive helps them make an informed assessment of buyers’ offers and negotiate grade and price more effectively. The program produces historical data that demonstrates the consistency of Canadian grains year over year. This data increases confidence in the quality, grading factors and specifications of the Canadian grain crop among current and potential buyers and processors. The fact that the Canadian Grain Commission is a government agency, independent of industry and producers, builds confidence in the Canadian grading system and the annual harvest reports. The detailed information on quality and consistency produced by the Harvest Sample Program also effectively assists the domestic grain industry in marketing Canadian grains.

The Harvest Sample Program supports the legislated responsibilities of the Canadian Grain Commission and contributes to the achievement of the Canadian Grain Commission’s strategic outcome. The materials obtained through the program are essential to the Canadian Grain Commission’s ability to recommend and establish grain grades and standards; implement a system of grading and inspection for Canadian grain; and conduct, sponsor and promote research about grain and grain products. The unofficial grades help participating producers with their marketing strategies. The annual harvest quality reports are widely used by grain companies, buyers, and processors to support the marketing of Canadian grains.

Program design and delivery

The current program design is efficient, cost-effective, and the best option based on the program’s purposes and beneficiaries. Other sampling methods would not meet the Canadian Grain Commission’s need for unique, individual producer-level samples, would not help producers market their grains and negotiate grade and price, or would be too expensive.

The design and delivery of the Harvest Sample Program is meeting the needs of the Canadian Grain Commission by providing an adequate supply of diverse sample materials for quality assurance and grain quality research. Other sources of sample materials do not meet the needs of the Canadian Grain Commission. Increasing the number of producers registered in the program and developing a way to target sample submissions to ensure growing regions and commodities are well represented would improve the delivery of the program.

The design and delivery of the Harvest Sample Program is very effective in addressing producers’ needs. Producers are for the most part very satisfied with the delivery of the program. Producers also suggest it is very important that the program remains free. If the Harvest Sample Program became a fee-for-service program, very few producers would participate and the program would not supply adequate sample materials for quality assurance and grain quality research. The producers surveyed suggested the Harvest Sample Program could be enhanced by:

  • revising the program’s online interface and emailing results to producers
  • providing the unofficial grade of a harvest sample to producers in a more professional looking printable format
  • allowing producers who own large farms to submit more than eight samples
  • including additional information in the unofficial grade and quality assessment given to producers

External recipients are very satisfied with the design and delivery of the Canadian Grain Commission’s annual harvest quality reports and information. These organizations and individuals reported that the content, format and delivery meet their needs well. International buyers value the ability to communicate directly with Canadian Grain Commission representatives to better understand the harvest quality reports and ask questions. Some industry associations and domestic grain companies and buyers suggested the program could be enhanced by obtaining more samples, targeting specific crops and regions, and publishing the reports earlier in the harvest season. Some international stakeholders indicated that combining the information produced by the Canadian International Grains Institute and the Canadian Grain Commission into one report and including more information on varietal end-use functionality would be useful.

Recommendations

The evaluation found that the Harvest Sample Program has been successful in achieving its objectives and is delivered in an efficient and cost-effective way. Possible opportunities to enhance the program were identified during the evaluation.

Recruit new program participants

As participating producers age, marketing and promotion efforts must ensure younger producers and those farming larger acreage are recruited to replace retiring producers. Current strategies such as tradeshows, communications through producer and industry associations, and social media should be continued. The Canadian Grain Commission should also look for new ways to promote and market the program to target the crops and growing regions that currently have lower participation. Possible strategies include using radio and print advertisements, direct contact with producers, local community events and enhanced partnerships with producer and industry associations.

Improve harvest quality information for producers

Improvements to the type and format of harvest quality information available to producers in order to increase their participation in the program should be considered. One possible improvement would be to include more technical factors in the unofficial grade and quality information provided to producers. The producers surveyed indicate that a more detailed explanation of the grading factors including falling number, dockage, moisture, hard vitreous kernels (HVK), bushel weight, fusarium and vomitoxin would enhance the program.

More sample envelopes could be provided to producers who own large farms and producers growing multiple varieties of the same crop. Providing registered producers with the ability to update which grains they intend to submit each harvest should also be considered.

Redevelop the online interface for producers

The online interface for producers could be improved. This could include a webpage where participating producers can login to update their sample preferences or notify the Canadian Grain Commission that they have retired, relocated or ceased operations. It could also allow producers to access their own results, compare their results year-over-year and to those of the regional and national composites. Producing a more professional, printable format of the unofficial grade and quality assessment should be considered to make the information even more useful for producers. To encourage participation, producers could be sent emails to remind them to submit samples, inform them that their results are ready, or to electronically deliver their results.

Improve tracking of producer participation

The number of producers removed from and added to the producer database each year could be included in the annual harvest survey internal reports issued to Canadian Grain Commission management in order to better track producer registration and participation in the program. If possible, the report could include the number of producers who submit samples. Current reporting only tracks the number of producers who receive sample kits and the total number of samples received.

Improve communication with stakeholders

The Canadian Grain Commission updates the wheat harvest information on a weekly basis, but these updates could be better communicated with the people who use this information. The Canadian Grain Commission could also increase communication with domestic stakeholders and international buyers to promote the Harvest Sample Program and to share the annual harvest quality reports.

Build partnerships to increase participation in eastern Canada

Continue to build partnerships with organizations in eastern Canada to improve the response rate among producers from Ontario and Quebec.

Work with the Canadian International Grains Institute to produce a comprehensive report

The possibility of producing a single comprehensive annual harvest quality report that combines the information produced by the Canadian Grain Commission and the Canadian International Grains Institute should be investigated. The US Wheat Associates’ harvest assessment report provides a model of a comprehensive national report that uses harvest quality data from multiple sources, using different collection methods.

Acknowledgements

We express our appreciation to grain producers and industry representatives, as well as staff and management of the Grain Research Laboratory and Industry Services for their assistance during this evaluation.

Program management

Dr. Stefan Wagener, Director – Grain Research Laboratory
Dr. David Hatcher, Program Manager

Audit and Evaluation Services Contact

Brian Brown, Chief Audit and Evaluation Executive

Date modified: