Evaluation of the Harvest Sample Program
Methodology
Evaluation approach
The evaluation relied on multiple lines of evidence including a document review, review of program data, interviews with key informants, surveys of producers and a literature review. By using multiple lines of evidence and triangulating the findings, the research methodology supported a comprehensive evaluation of the Harvest Sample Program.
Document review
The document review addressed evaluation issues related to the relevance and performance of the Harvest Sample Program. A total of 47 documents were reviewed as part of the evaluation (Appendix 5), including:
- legislation including the Canada Grain Act
- the Program Logic Model
- annual harvest quality reports
- speeches from the Throne
- Departmental Performance Reports
- Reports on Plans and Priorities
- program expenditure reports
- past reviews and evaluations of the Harvest Sample Program
Review of program data
Program data was reviewed to obtain information related to the relevance and performance of the Harvest Sample Program. Data analyzed as part of the evaluation included the number of registered producers in the producer database, previous customer satisfaction survey data, annual program summary reports which include producer participation rates, and performance data on visits to the program webpage and Canadian Grain Commission annual harvest quality report webpages.
Key informant interviews
A total of 83 interviews were conducted with key informants between October 2015 and February 2016 to address evaluation issues dealing with the relevance and performance of the program. Respondents were identified in consultation with the Project Steering Committee and sent an email explaining the purpose and timing of the interviews, a letter of introduction from the Canadian Grain Commission, and a copy of the interview questionnaire. Interviews were conducted by telephone and the responses were compiled and analyzed using Excel. Due to constraints scheduling telephone interviews with time differences and language barriers, some international buyers were provided the opportunity to submit their feedback via a written questionnaire. The number of respondents interviewed and the response rate for each stakeholder group is provided in Table 3.1.
| Group | Initial sample | Valid sample | Target | Actual | % of Target | Response rate | Est. sample errorFootnote 1 |
|---|---|---|---|---|---|---|---|
| Canadian Grain Commission representatives | 28 | 28 | 20 | 26 | 130% | 92.8% | 5.2% |
| Associations | 49 | 47 | 15 | 19 | 127% | 40.4% | 17.5% |
| Domestic grain companies and buyers | 44 | 40 | 15 | 21 | 140% | 52.5% | 14.9% |
| International buyers and processors | 69 | 63 | 20 | 17 | 85% | 26.9% | 20.5% |
| Total | 190 | 178 | 70 | 83 | 119% | 46.6% | 7.9% |
Surveys of producers
Between November and December 2015, surveys were conducted with 1,207 producers who participated in the Harvest Sample Program, and 125 producers who registered for but have not participated in the program. The survey of participating producers was designed to obtain information from producers who have submitted samples regarding the use and benefit of the unofficial grade and quality assessment, their motivation for participating, and their degree of satisfaction with the services received. The survey of producers who registered but did not participate was designed to obtain information regarding their motivation for registering and the reasons why they have never submitted samples.
Survey respondents were sourced from the Harvest Sample Program producer database of 6,727 active producers. The database included 2,449 producers without an email address and 4,278 with an email address, 193 of whom had multiple registries and were excluded. The remaining 4,085 producers with emails were sent invitations to participate in the survey. In addition, 200 randomly selected producers with no email address were mailed a hard copy of the survey. Participants were sent pre-survey communication which included a letter of introduction and a copy of the questionnaire. A total of 3 email reminders were sent to producers with an email address and all participants received at least one telephone reminder. Updated email addresses were obtained for producers whose initial invitation had bounced and new invitations were reissued. Survey responses were coded and cleaned using Excel and summarized according to evaluation issue.
Table 3.2 outlines the number of survey completions and response rate for each cohort.
| Group | Initial sample | Invalid or bounced |
Valid sample | Target | Completed | % of target | Response rate |
Margin of errorFootnote 1 |
|---|---|---|---|---|---|---|---|---|
| Participating producers |
4,285 | 413 | 3,872 | 1,000 | 1,207 | 121% | 31.2% | 2.3% |
| Non participating producers |
N/A | N/A | N/A | 150 | 125 | 83% | N/A | N/A |
Literature review and comparative analysis of similar programs
The literature review and comparative analysis were conducted to obtain information related to the need for the Harvest Sample Program and to compare the effectiveness and efficiency of the program with similar programs in Canada and in other jurisdictions. Other programs were identified in consultation with the Evaluation Steering Committee, and included:
- the Canadian International Grains Institute
- Grain Farmers of Ontario
- the US Wheat Association
- the Australian Grain Growers Association / Australian Export Grains Innovation Centre
- FranceAgrimer/Arvalis
- the United Kingdom’s Agriculture and Horticulture Development Board’s Cereals and Oilseeds Division
A total of 50 websites, journal articles and grey literature sources were reviewed (Appendix 6).
Methodological considerations
Some of the limitations of the evaluation and the strategies employed to mitigate these limitations are as follows.
There is a potential response bias in the findings of the key informant interviews. This limitation was mitigated by:
- clearly communicating the purpose of the evaluation, its design and methodology, and strict confidentiality of responses with respondents
- cross-checking the responses with those of other stakeholder groups
- using multiple lines of evidence
- triangulating the evaluation findings
A relatively small number of completed interviews with international buyers and processors of Canadian grains makes it more difficult to ensure the results are significant, and to generalize the findings to the population of international stakeholders who use the annual harvest quality reports and other information.
"Page details"
- Date modified: