Evaluation of the Quebec Economic Development Program (QEDP)

Download the printer-friendly version (PDF)

About this publication

Publication author : Canada Economic Development for Quebec Regions

ISSN number : 2561-8377

Catalog number : Iu90-1/20E-PDF

Publish date : August 24, 2018

Summary : This report is an evaluation of the Quebec Economic Development Program (QEDP) implemented by Canada Economic Development for Quebec Regions (CED) in April 2012. The evaluation covers the period from April 2012 to March 2016.

Table of Contents

  1. Summary
  2. Introduction
  3. 1. Summary Description of the QEDP
  4. 2. Evaluation Methodology
  5. 3. Relevance
  6. 4. Effectiveness
  7. 5. Efficiency
  8. 6. Conclusion
  9. 7. Action Plan

Summary

This report is an evaluation of the Quebec Economic Development Program (QEDP) implemented in April 2012 by Canada Economic Development (CED) for Quebec Regions. In this report, the program’s relevance, effectiveness and efficiency is assessed, as recommended by the Policy on Results.

The evaluation covers the period between April 1, 2012, and March 31, 2016, during which CED spent $658.3 million in grants and contributions on 2,098 projects under the QEDP. These projects include those under the Local Investment Initiative (LII), Initiative to Rehabilitate Water Crossings on Wildlife and Multi resource Roads and Strategic Initiative to Combat the Spruce Budworm Outbreak in Quebec, as well as the Bellechasse Pipeline project.

This summary presents key findings in connection with eight evaluation questions, as well as recommendations and the resulting action plan.

Relevance

1. Did the QEDP meet needs? How have these needs changed over time?

2. How does the QEDP complement or overlap with other interventions?

3. How is the QEDP aligned with government priorities?

Effectiveness

4. What is the degree of usefulness and effectiveness of performance measurement?

5. To what extent has the QEDP been effective in achieving the expected results?

6. What is the leverage effect and what are the other impacts of the QEDP?

Efficiency

7. Were there any problems and facilitating factors during the various phases of the QEDP life cycle?

8. How can QEDP efficiency be improved?

Recommendations

Recommendation 1: Every five years, CED should establish and approve, in a timely manner, targets for QEDP outcome indicators.

Recommendation 2: To better assess client satisfaction and alignment with needs, CED should document its interactions with all project proponents, including those that received an informal refusal from CED.

Recommendation 3: CED should ensure that:

  1. Its results chains are always consistent with the design of funded activities;
  2. Its performance indicators are always reasonably linked to its activities.

Recommendation 4: To improve the client experience, CED should continue to simplify its administrative processes and work in collaboration with other funders to minimize the collective administrative burden on clients.

Recommendation 5: To better meets the needs of businesses and regions in Quebec, CED should ensure that:

  1. Its service delivery model and eligibility criteria for the QEDP are tailored to the specifications of the target clientele;
  2. Its advisors have or can easily access the specialized expertise they need to analyze projects and better support proponents.

Recommendation 6: To optimize the efficiency of implementing the QEDP, CED should:

  1. Continue to integrate various departmental guidance tools;
  2. Document the cost effectiveness of using various types of contributions based on risk level and the assistance amounts allocated;
  3. Document the efficiencies achieved further to establishing new business processes.

Link to the Action Plan

Introduction

This report is an evaluation of the Quebec Economic Development Program (QEDP) implemented by Canada Economic Development (CED) in April 2012.

The evaluation covers the period from April 2012 to March 2016. It deals with the relevance, efficiency and effectiveness of the Business Development and Regional Economic Development pillars and the four initiatives under the Strengthening Community Economies pillar, namely the Local Investment Initiative (LII), the Initiative to Rehabilitate Water Crossings on Wildlife and Multi-resource Roads, the Strategic Initiative to Combat the Spruce Budworm Outbreak in Quebec, and the Bellechasse Pipeline project.

In addition to establishing formal accountability, this evaluation is a decision support tool designed to improve QEDP interventions.

This report is divided into the following seven sections:

  1. Summary Description of the QEDP
  2. Evaluation Methodology
  3. Relevance
  4. Effectiveness
  5. Efficiency
  6. Conclusion
  7. Action Plan

1. Summary Description of the QEDP

1.1. Foundations

By implementing the QEDP in 2012, the objective was to support Quebec regions in dealing with the economic challenges resulting from various factors, including:

To overcome these challenges, CED implemented the QEDP, which consists of three pillars:

  1. The Business Development pillar aims to support businesses throughout their life cycle. Businesses are recognized as a driver of economic development, stimulate economic activity and create jobs in communities. The purpose of this pillar is to support the renewal of Quebec’s pool of businesses through support for business creation and startup. It seeks to increase their performance by financially supporting their expansion, productivity gains, innovation and technology transfer, commercialization and export, and network structuring activities.
  2. The Regional Economic Development pillar aims to financially support the creation of development strategies, the construction of community economic facilities and the promotion of regional assets to attract foreign direct investment and tourists.
  3. The Strengthening Community Economies pillar aims to implement national programs or temporary, targeted initiatives intended to strengthen community economies and increase Quebec’s economic growth. Temporary initiatives that have received dedicated funds are evaluated separately.

1.2. Eligible clients

The main clients eligible for QEDP support are:

1.3. Logic model

The QEDP logic model is presented in Table 1.1. From bottom to top, this model identifies the following components:

Table 1.1
QEDP logic model
Strategic Outcome Quebec’s regions have a growing economy
Pillars Business Development Regional Economic Development Strengthening Community Economies
Final outcome
The pool of businesses in Quebec is renewed
Quebec’s businesses are competitive
Quebec regions have a stronger economic base Quebec communities have stronger economies
Focus Entrepreneurship support Business performance Regional engagement Regional investment Targeted and temporary support
Intermediate outcomes
Businesses are created or transferred Businesses improve their performance Communities take charge of their economic development Quebec’s regions attract or generate investment Communities receive temporary support to stabilize or strengthen economies
Components Business creation and startup Business succession and transfer Productivity and expansion Innovation and technology transfer Commercialization and exports Network structuring Economic development strategy Community economic facilities Promotion of regional assets Temporary and targeted initiatives
Immediate outcomes
Entrepreneurs or businesses are engaged in business startup Entrepreneurs are engaged in business transfer Businesses optimize their production capacity Businesses innovate or increase their capacity to innovate Businesses commercialize or increase their capacity to commercialize Businesses have structured networks Community players are engaged in a mobilization process Regions engage in a process to improve or obtain community economic facilities Regions develop their assets - Initiatives with QEDP core funds included in this evaluation (for example, Local Investment Initiative and Spruce Budworm Initiative)

- Initiatives with dedicated funds evaluated separately (for example, Lac-Mégantic Initiative and Chrysotile Initiative)
Outputs Total number of projects per component
Total value of projects per component
Activities Direct project funding in connection with eligible activities with target proponents (direct assistance)
Indirect funding from intermediary groups to be involved in eligible activities with the target population (indirect assistance)
Inputs Total number of full time equivalents (FTEs) and employee benefit plans (EBPs)
Grants and contributions (G&C) amounts
Operating and maintenance costs, costs of premises

Source: QEDP Performance Measurement Strategy (2012).

1.4. Intervention approach

The QEDP is an “on demand” program whereby proponents submit projects to the business office in their region, which are then analyzed against government, departmental and business office priorities.Footnote 2

CED assistance is typically provided in two ways:

1.5. Financial resources

Table 1.2 shows the QEDP budget envelope for the period of 2012-2013 to 2015-2016. Excluding initiatives for which CED obtained dedicated one-time funding, the QEDP’s spending totals $740.2 million, including $658.3 million in grants and contributions and $81.9 million in operating costs.

1.6. Summary overview of interventions

Between April 1, 2012, and March 31, 2016, the QEDP was used to support 2,098 projects, for a total of $658.3 million in G&C assistance. The number of projects and the annual G&C expenditures for their implementation are also presented in Table 1.2.

Table 1.2
QEDP expenditures from 2012–13 to 2015–16
Fiscal Year Operations (millions of dollars) Grants and Contributions (millions of dollars) Total Expenditures (millions of dollars)
Business Development (1,740 projects)
2012–2013 21 109.4 130.4
2013-2014 17.9 129.7 147.6
2014-2015 14.8 131.7 146.5
2015-2016 15.5 132 147.5
Regional Economic Development (219 projects)
2012-2013 4.8 39.2 44
2013-2014 2.9 36.2 39.1
2014-2015 2.1 32 34.1
2015-2016 2.3 31.3 33.6
Strengthening Community Economies (139 projects, excluding initiatives where CED obtained dedicated funds)
2012-2013 n/a n/a n/a
2013-2014 n/a n/a n/a
2014-2015 0.2 5.2 5.9
2015-2016 1.1 11.6 12.7
Annual total for the three pillars (2,098 projects)
2012-2013 25.8 148.6 174.4
2013-2014 20.8 165.9 186.7
2014-2015 17.1 168.9 186
2015-2016 18.2 174.9 193.1
QEDP total 81.9 658.3 740.2

Source: 2012–13 to 2015–16 reports to Parliament.

1.7. Program implementation context

QEDP implementation falls within an ever changing operational and organizational context that can have both a direct and an indirect impact on how the program is carried out. Those contextual aspects and their impact on the program are shown in Table 1.3.

Table 1.3
QEDP implementation context
Context Implementation environment of the program Impacts on the program
Federal government context Change in governmental priorities in 2015. Adaptation of CED priorities to be aligned with government priorities
Provincial government context Re-organization of the Centres locaux de développement (CLD) in 2014-2015. Larger draw on CED resources to deal with this transition
Fund transfer of the Ministère de l’Économie de la Science et de l’Innovation (MESI) to the Investissement Québec in 2015.
Departmental context Deficit reduction action plan in 2012 including a 20% decrease in the number of FTEs, the redistribution of the territories served by the business offices and the consolidation of the Laval-Laurentides-Lanaudière business offices, of the Montérégie region and the Nord-du-Québec region, as well as the repatriation of communication activities to the head office in Montreal. Pressure on the organization of the work to implement the QEDP
Implementation of four initiatives funded with CED's core budget: Local Investment Initiative; Initiative to restore watercourse crossings on wildlife and multi-resource roads ($ 6 millions, 2014-2017); Initiative to combat the spruce budworm outbreak ($ 6 millions, 2014-2018); Bellechasse Pipeline project ($ 17.5 million, 2012-2017). Impact on resources dedicated to the Business Development and Regional Economic Development sub-programs
Implementation of the Canada 150 Community Infrastructure Program, with a budget of $ 31.2 million.

2. Evaluation Methodology

This section presents the QEDP evaluation approach, which includes its terms of reference, related issues, its governance, the adopted strategy and the limitations of the methodology.

2.1. Evaluation terms of reference

This evaluation is part of the five year review of grants and contributions programs, as required by the Financial Administration Act (section 42.1). It looks at relevance, effectiveness and efficiency.

2.2. Monitoring committee

A monitoring committee was created to assist with the evaluation and was tasked with commenting on the various deliverables (evaluation framework, interview guides, preliminary findings, reports), facilitating access to program data and providing advice at various stages of the evaluation process in order to maximize usefulness for CED.

This committee is composed of the evaluators, two representatives of the responsibility centres, and the chief of evaluation of the Atlantic Canada Opportunities Agency (ACOA).

2.3. Scope of evaluation

The evaluation examines all the projects supported through the QEDP core budget and excludes initiatives for which CED obtained dedicated funds.

This evaluation therefore looks at the Business Development and Regional Economic Development pillars and the four initiatives under the Strengthening Community Economies pillar, implemented with the QEDP core budget, namely the Local Investment Initiative (LII), the Initiative to Rehabilitate Water Crossings on Wildlife and Multi-resource Roads, the Strategic Initiative to Combat the Spruce Budworm Outbreak in Quebec, and the Bellechasse Pipeline project.

Initiatives funded using temporary budget envelopes are dealt with in separate evaluations.

The scope of this evaluation varies depending on the issues examined:

2.4. Data collection methods

In order to provide as thorough an answer as possible to the evaluation questions, a number of collection methods and data sources were used. Wherever possible, preference was given to a combination of qualitative and quantitative methods. The choice of methods was determined based on their relevance and reliability, data availability and related costs. Table 2.1 summarizes the methods used.

2.5. Evaluation limitations

The evaluation design took into account the limitations of the data sources used. Thus, when the evaluation strategy was being developed, various measures presented below were applied to draw conclusions based on factual findings and a thorough methodology.

Targets: The performance measurement strategy was updated between April 2015 and March 2016, but considering that the Policy on Results was supposed to be coming out shortly, the recommended changes, including the targets and replacement of some indicators, were not approved. When they were available, the evaluation nonetheless used these targets.

Data on business performance: The dates on which the financial statements of businesses were released did not always match the dates during which CED data were collected, meaning that information was not always available at the time of the evaluation.

Scope of Activity Information System (AIS): This tool is used to enter the amount of time dedicated to managing each project phase. For the purposes of simplification, it was decided that half the business office advisors would compile the time spent managing projects in their portfolio for the first six months of the year, and then the other half of the advisors would do the same for the six remaining months. While this method compares the time spent managing various types of projects, it is not sufficiently accurate to estimate actual time.

Table 2.1
Data collection methods used
Collection tools and objectives Sources
Internal data and documentation: Document the program design and implementation
  • Financial data
  • QEDP design sheets
  • 2012 2017 Strategic Framework
  • Regional strategies
  • Intervention parameters
  • Hermès
Literature review: Validate alignment with government priorities
  • Speech from the Throne
  • Federal Budget
  • Minister’s Mandate Letter
Semi-directed interviews with internal and external stakeholders: Assess QEDP relevance, effectiveness and efficiency from the point of view of various stakeholders
  • 29 internal stakeholders : 4 senior managers; 5 representatives of the head office; 20 advisors from the business offices
  • 30 external stakeholders: 7 funders, 14 local stakeholders; 7 clients
Telephone surveys of proponents:
Assess QEDP relevance and effectiveness from clients’ point of view.
  • Telephone survey conducted from May 18 to June 30, 2016, of 1,328 businesses and NPOs that received CED funding

Results from intermediary groups: CED sends an annual online survey to clients of intermediary groups (IGs) to measure the impact of services received from IGs. Only businesses that conducted a high intensity activity are contacted. Businesses that only conducted a basic activity, such as information services and advice of less than three hours, are not considered in the survey.

Response rate for online surveys of IG clients: The online survey response rate is around 29%. Thus, results were provided as an indication of the degree of satisfaction with the alignment between the services businesses received and their needs, and the impact of those services on the performance of their business. Results are not representative of the overall IG clients.

Statistics Canada study on the impact of CED interventions: Although the method used in this study is most thorough to report on the impact of CED interventions compared to a control group, the reference period for this study (2001 to 2013) is different from that of the current evaluation. This study is nonetheless relevant for the requirements of this evaluation since the types of projects funded are similar to those funded under the QEDP.

Client satisfaction was evaluated: using the database of clients whose projects received financial assistance. Since CED does not document informal refusals, the evaluation could not cover the perspective of other businesses that did not receive assistance.

Collective economic equipment: the evaluation methodology of this type of project does not allow for measuring the impacts of public utility projects for the following two reasons: 1) the long delay for the equipment to be operational; and 2) current indicators are insufficient to capture all potential impacts of such equipment.

3. Relevance

Questions explored and underlying indicators

Did the QEDP meet needs? How have these needs changed over time?

  • 3.1. Nature and evolution of needs
  • 3.2. Potential impacts of absence of QEDP
  • 3.3. Degree of consistency between distribution of allocated funding and needs
  • 3.4. Degree of alignment of the QEDP delivery model with business and regional needs

How does the QEDP complement or overlap other interventions?

  • 3.5. Degree of overlap and complementarity between the QEDP and other comparable programs
  • 3.6. Degree of coordination between stakeholders

How is the QEDP aligned with government priorities?

  • 3.7. Degree of alignment between government priorities and CED priorities for the QEDP
  • 3.8. Degree of alignment between departmental priorities and QEDP implementation

Highlights

  • The program has met the needs for which it was implemented in 2012: A comparison of the needs underlying the QEDP and current needs in the regions shows that the issues and challenges justifying the program’s continuation remain.
  • The program is vital for most projects: Without CED’s financial assistance through the QEDP, most projects would not have been carried out.
  • The value of the assistance approved is higher in regions with strong economic potential where the number of financial assistance applications is higher: Although the value of the assistance approved is proportional to the demographic weight of the regions.
  • The QEDP delivery model segmented by component does not always meet business and regional needs: The many eligibility criteria specific to each QEDP component make the interventions less agile and, in some cases, can increase the time it takes to deliver them to clients.
  • The program does not overlap with other interventions, but the large number of funders is a challenge for some clients: There is little overlap because funding needs are high and financial partners are bound by assistance- stacking rules. Nonetheless, the large number of funders risks creating confusion or even causing an administrative burden for clients.
  • Coordination between stakeholders is observed and appreciated by clients: Most QEDP clients have noted some coordination between stakeholders on the ground. However, according to the stakeholders interviewed, there is still progress to be made.
  • CED’s priorities for the QEDP are aligned with government priorities: During the period under review, departmental priorities were in line with those of the government, and the objectives underpinning the QEDP components are consistent with government priorities set out in official documents.
  • The extent of the assistance awarded under the components of the QEDP is generally consistent with Departmental priorities with some nuances observed under the Promotion of Regional Assets component, mainly due to projects receiving recurring funding in some regions.

3.1. Nature and evolution of needs underpinning the QEDP

Quebec’s regional economic conditions from 2012 to 2016, the needs expressed by clients and the challenges identified by various stakeholders were analyzed. Two key findings stems from this analysis:

Finding no. 1: While the economic conditions of Quebec have generally improved since the implementation of the QEDP, regional disparities remain. Thus, the need to stimulate innovation among small businesses is still present.

This finding is based on changes in six economic indicators,Footnote 3 presented in Table 3.1, namely gross domestic product (GDP) at basic prices, economic diversity, the number of jobs, household primary income per capita, the Economic Development Index (EDI)Footnote 4 developed by CED and the innovation rate. Six observations associated with these indicators thus arise:

  1. GDP at basic prices: Quebec’s resource, manufacturing and urban regionsFootnote 5 have experienced uneven economic performance, despite an overall improvement for Quebec as a whole from 2010 to 2015.
    Table 3.1 shows the trends for each type of region. Urban regions have stronger performance and higher growth due notably to growth in service industries. The performance of manufacturing regions reflects that of Quebec as a whole, while that of resource regions has been fairly weak and steady since 2010, particularly due to the drop in commodity prices and the end of hydroelectric projects.
  2. Economic diversity:Footnote 6 Economic diversity is uneven between the regions served by the QEDP, even though Quebec compares favourably overall to other provinces owing to greater employment resilience.
    Resource regions had a low degree of diversity (29.8 out of 100) in 2016 compared to manufacturing and urban regions, where this index was 85.3 and 84.9 out of 100, respectively. Since the diversity index is slow to evolve, no significant changes have been observed since the start of the QEDP.
  3. Number of jobs:Footnote 7 From 2011 to 2016, 157,500 jobs were created in Quebec. Of this number, roughly two thirds were created in urban regions and one third, in manufacturing regions. A slight loss in jobs was observed in resource regions during this period.
    This weak performance in resource regions is due in part to low diversity in their economy, which was not capable enough to withstand the drop in commodity prices. Added to this is the aging of the population since the 2006 census. This aging is occurring at a faster pace in manufacturing and resource regions, where the proportion of people aged 65 and over has increased by 5.9 and 5.4 points, respectively, in 10 years.
  4. Primary household income:Footnote 8 The average rise in primary household income in Quebec was 2.7% between 2010 and 2015, with a higher increase in resource regions.
    However, urban regions have the highest primary household income per capita, followed by manufacturing regions. Resource regions have the strongest income growth.
  5. Economic Development Index (EDI): Of the 104 MRCs, 71 were identified as having low economic potential in 2016, compared to 68 in 2012.
    Developed by CED, this index is an aggregate measure of the economic development of MRCs in Quebec. It is used mainly to adjust CED assistance based on the economic potential of Quebec’s regions. According to the EDI, the number of MRCs with low potential increased from 68 to 71 between 2012 and 2016.
  6. Innovation: Between 2012 and 2014, approximately 50.9% of businesses in Quebec introduced at least one type of technological innovation (for example, products and processes).
    According to the advanced technologies survey conducted by the Institut de la statistique du Québec, L’utilisation des technologies de pointe par les entreprises du Québec (2014), most businesses that use such technologies have 250 employees or more and operate in the manufacturing sector, even though there is still a need to boost innovation in smaller businesses. Technological innovation excludes organizational and business innovation.

Finding no. 2: The challenges that the QEDP aims to overcome typically have a long term scope, so the need for intervention is still there. Also, the areas of intervention targeted by the QEDP are still relevant to variable degrees depending on the components and regions.

Table 3.2 summarizes, for each QEDP intervention component, the needs identified following interviews with CED stakeholders.Footnote 9 However, the needs identified by QEDP clients vary from one intervention component to the next:

Lastly, the needs expressed by QEDP clients vary depending on the region to which they belong:

Table 3.1
Economic indicators in Quebec by regional blocks in 2010 and 2015
Indicator All of Quebec Resource regions Manufacturing regions Urban regions
2010 2015 2010 2015 2010 2015 2010 2015
GDP at basic prices (B$) 305.5 351.1 24 26 121.1 139.2 160 185.3
Economic diversity (out of 100) n/a n/a 32.9 29.8 86 85.3 85.5 84.9
Number of jobs (thousands) 3 975.6 4 133.1 254.8 249.1 2 013.6 2 070.5 1 707.3 1 813.7
Primary household income ($) 28 301 32 317 24 021 28 114 27 715 31 673 29 730 33 762
Number of MRCs with low economic potential* 68 71 26 26 32 35 10 10
Innovation rate
(% of businesses that introduced at least one type of innovation)**
50.9% n/a n/a n/a n/a n/a n/a n/a

* The EDI calculations date back to 2012 and 2016.

** The data cover the period from 2012 to 2014.

Source: Government of Quebec (2017). Portrait économique des régions du Québec 2017 and CED compilation of EDI data.

Table 3.2
Assessment of needs identified by QEDP component
QEDP component Needs identified by stakeholders
Business creation and startup
  • Needs for funding and guidance for SMEs in startup phase
  • Need for more funding and seed capital
Business succession and transfer
  • Need for entrepreneurial succession
  • Little venture capital in surrounding areas
Productivity and expansion
  • Need to integrate new technologies
  • Need to support equipment modernization
Innovation and technology transfer
  • Need to encourage SMEs to invest more in R&D
  • Need for more commercialization of innovations
  • Need for more collaboration between universities and businesses
Commercialization and exports
  • Need for SMEs to increase their commercialization capacity in order to improve their competitive edge
Network structuring
  • Need to raise awareness among businesses so that they increase their participation in sectoral groups / clusters
Development strategy
  • Need for mobilization to formulate a consistent development framework for regional diversification
Community economic facilities (CEFs)
  • Need for more new CEFs that make various communities more attractive
  • Need to expand the pool of businesses to justify CEF funding
Promotion of regional assets
  • Need to be more competitive to attract tourists
  • Need to develop the significant potential of regional assets
  • Need to expand tourism spinoffs beyond large urban centres

Source: Interviews with CED stakeholders.

3.2. Potential impacts of absence of QEDP

Program clients, CED stakeholders and their partners were asked about the impact that the absence of QEDP financial assistance would have on supported projects and regions. The four following findings were made:

Finding no. 3: Out of the 1,328 clients interviewed, 63% reported that they would have been unable to carry out their project without financial assistance from CED.

With respect to QEDP components, the results shown in Figure 3.1 indicate that the proportion of clients dependent on financial assistance from CED is generally higher for clients of the Innovation and technology transfer (79%) and Promotion of regional assets (84%) components.

As for the type of proponents, 83% of NPOs and 50% of businesses supported reported that they would have been unable to implement their projects without the assistance they received.

Finding no. 4: Of the 463 clients who responded that they would have been able to carry out their project without financial assistance from CED, more than half stated that the project would not have been implemented within the same timeframe (57%) or with the same scope (58%).

Figures 3.2 and 3.3 show that most clients able to complete their project without CED financial assistance would have been unable to do so within the same timeframes or with the same scope.

Finding no. 5: Most of the clients interviewed (52%) requested financial assistance from CED because the funding conditions offered are viewed as being more advantageous than those of other programs.

This finding is illustrated by Figure 3.4, which shows the main reasons why clients of various components requested financial assistance from CED.

Figure 3.1
Potential project completion, without CED assistance? (% of clients interviewed; N=1,328)
description follows

Question: Would you have been able to carry out your project without CED assistance?
Source: Telephone survey of clients, 2016.

Description of figure 3.1
Figure 3.1. Project completion possible without CED assistance?
Components Project completion possible without CED assistance?
(percentage of clients interviewed, N = 1,328)
No Yes Don’t know/refusal
Business creation and start-ups 51 % 45 % 4 %
Productivity and expansion 53 % 44 % 3 %
Innovation and technological transfer 79 % 20 % 1 %
Commercialization and export 63 % 36 % 1 %
Promotion of regional assets 84 % 16 % 0 %
Question: Would you have been able to carry out your project without CED assistance?
Source: Telephone survey of clients, 2016.
Figure 3.2
Potential project completion, within same timeframes, without CED assistance? (% of clients able to realize their projects without the QEDP; N=463)
description follows

Question: Would you have been able to carry out your project within the same timeframes, without CED assistance?
Source: Telephone survey of clients, 2016.

Description of figure 3.2
Figure 3.2. Project completion possible within the same timeframes without CED assistance?
Components Project completion possible within the same timeframes without CED assistance? (percentage of clients interviewed, N = 463)
No Yes
Business creation and start-ups 64 % 32 %
Productivity and expansion 50 % 47 %
Innovation and technological transfer 63 % 37 %
Commercialization and export 78 % 22 %
Promotion of regional assets 58 % 42 %

Question: Would you have been able to carry out your project within the same timeframes, without CED assistance?

Source: Telephone survey of clients, 2016.

Figure 3.3
Potential project completion, with same scope, without CED assistance? (% of clients able to realize their projects without the QEDP; N=463)
description follows

Question: Would you have been able to carry out your project with the same scope, without CED assistance?
Source: Telephone survey of clients, 2016.

Description of figure 3.3
Figure 3.3. Project completion possible within the same scope without CED assistance?
Components Project completion possible within the same scope without CED assistance? (percentage of clients interviewed, N = 463)
No Yes
Business creation and start-ups 68 % 32 %
Productivity and expansion 51 % 48 %
Innovation and technological transfer 58 % 26 %
Commercialization and export 71 % 28 %
Promotion of regional assets 83 % 17 %

Question: Would you have been able to carry out your project within the same scope, without CED assistance?

Source: Telephone survey of clients, 2016.

Figure 3.4
Proportion of clients by component according to the reason they requested financial assistance under the QEDP (N=1,328)
description follows

Source: Telephone survey of clients, 2016.

Description of figure 3.4
Figure 3.4. Proportion of clients by component according to the reason they requested financial assistance under the QEDP
N=1,328 Business creation and start-ups Productivity and start-ups Innovation and technological transfer Commercialization and export Promotion of regional assets Other
Funding conditions deemed advantageous 45,5 % 60,6 % 55,1 % 58,6 % 19 % 41 %
Support necessary for financial package 48,5 % 34,3 % 36,7 % 34,3 % 75 % 46 %
Pre-conditions for CED assistance 4 % 4 % 5,1 % 3 % 4 % 9 %
Funding refused by other previously contacted funders 1 % 0 % 3,1 % 3 % 1 % 4 %
Other reasons 1 % 1 % 0 % 1 % 1 % 0 %
TOTAL 100 % 100 % 100 % 100 % 100 % 100 %

Source: Telephone survey of clients, 2016.

Finding no. 6: CED financial assistance also enabled 40% of the clients interviewed to complete the financial package for their project. Additionally, it provided regions with low economic potential with vital support for the implementation of 772 projects between April 2012 and March 2016.

The highest proportions of clients who mentioned this benefit of QEDP financial assistance in terms of the financial package are in regions that are generally far from large urban centres, such as Nord du Québec (54%), Gaspésie Îles de la Madeleine (53%) and Bas Saint Laurent (49%).

These results are partly due to the smaller number of financial partners in communities far from large urban centres. In these communities, most of the clients in question are NPOs.

3.3. Degree of consistency between distribution of allocated funding and needs

The alignment between the assistance amounts approved between 2012 and 2016, the regions’ demographic weight, their economic potentialFootnote 10 and the number of projects carried out was evaluated.

Finding no. 7: The distribution of approved assistance is, overall, consistent with the demographic weight of Quebec regions, although it is not a criterion for the choice of project. The assistance amounts and the number of projects carried out are higher in regions with strong economic potential where the number of financial assistance applications is higher.

Under its mission, CED pays particular attention to regions with low economic potentialFootnote 11. Table 3.3, which shows the amounts of the assistance authorized from April 2012 to March 2016 based on the type of MRC, reflects that objective: Based on the share of the working population, the financial assistance from CED awarded to projects in MRCs with low potential is relatively greater that what was awarded in the other MRCsFootnote 12.

Since the number of applications is higher in MRCs with strong economic potential, the assistance amounts awarded to the projects supported there are higher than those in MRCs with low potential.

Finding no. 8: The approved assistance amounts are not always consistent with the components where CED assistance was deemed more vital by clients interviewed.

This finding is illustrated in Table 3.4 which shows, for each QEDP component, the assistance approved and the proportion of clients who agree or disagree about the financial assistance received being vital to the implementation of their project. In addition, the proportion of beneficiary NPOs judging that CED assistance is essential (85%) is higher than that of beneficiary SMEs (45%) for all sectors combined.

Table 3.3
Alignment of approved assistance to categories of MRCs with strong and low economic potential
EDI MRC rank according to EDI (2016) Total assistance approved from 2012 to 2016 (millions of dollars) No. of approved projects Average assistance per capita (millions of dollars) 2015 Working population
High potential 1-26 409.4 738 136 3,010,090
27-33 77.4 155 467.2 165,678
Sub total 1-33 486.9 893 153.3 3,175,768
Low potential 34-52 76.7 202 19 401,627
53-78 86.4 251 246.1 351,019
79-104 45.7 185 225.1 203,062
Sub total 34-104 208.8 638 218.5 955,709
Total 695.7 1,531 168.4 4,131,478

* Working population: population from age 15 to 64

Source: Internal compilation.

Table 3.4
Correspondence between approved assistance by component and proportion of respondents who deemed CED assistance vital
QEDP component Assistance approved Would not have completed project without CED assistance
Rank M$ Rank %
Business creation and startup 4 76.7 9 54
Business succession and transfer 9 0.2 5 77
Productivity and expansion 1 266.2 8 55
Innovation and technology transfer 3 103.3 3 81
Commercialization and exports 5 56.9 7 64
Network structuring 7 14.1 4 78
Economic development strategies 8 1.7 1 90
Community economic facilities 6 21.1 6 68
Promotion of regional assets 2 185.5 2 84

Source: Internal compilation.

3.4. Degree of alignment of the QEDP delivery model with business and regional needs in Quebec

The alignment of QEDP program and funding parameters with business and regional needs is assessed using information gathered from QEDP clients as well as CED stakeholders and partners.

Four findings were made:

Finding no. 9: According to some CED stakeholders, the many eligibility criteria make the interventions less agile and, in some cases, can increase the time it takes to deliver them.

According to some CED stakeholders, there are too many eligibility criteria under each component, which makes the interventions not agile enough to address regional specificities. Those criteria involve the following risks:

Finding no. 10: In order to better align the QEDP with changing business and regional needs, CED stakeholders suggested improvements that include revising the eligibility criteria for the Business succession and transfer and Promotion of regional assets components.

These improvements, shared by CED stakeholders during interviews, are detailed below:

  1. With respect to repayment terms, extension of the repayment period based on the nature of the funded activities would reduce the financial pressure on projects that span, for instance, a number of years or that are carried out by seasonal businesses.
  2. As for intervention component requirements, relaxing the operational guidelines, particularly for the Business succession and transfer component, would make it easier to deal with buyers. Such relaxing of the requirements could, for example, take the form of greater program openness to other business lines, including primary processing activities, or even less restrictive funding in the form of share capital.
  3. For the Promotion of regional assets component, recurring assistance for the same festivals and sporting events could be reviewed in light of the changing context in which they operate. At the same time, the possibility of CED supporting the renewal of tourist products could be explored.
  4. In terms of assistance type, more non repayable or conditional contributions (type D) would enable SMEs to fund diagnostic activities in order to better define their needs and plan their market expansion.
  5. As for risk taking,Footnote 13 funding could be provided to high risk projects, such as those that usually fall under the Business creation startup component.

Finding no. 11: For 76% of the QEDP clients interviewed, the funding conditions offered were in line with their needs.

This finding is based on the responses gathered during the telephone survey of interviewed clients regarding their satisfaction with CED funding conditions (Table 3.6). Thus:

The requests for changes that were not accepted (4% of the companies interviewed), involved increasing the funding and the eligibility of some expenditures.

Table 3.5
Adaptation of funding conditions to needs
Responses (N = 1,328) % of clients interviewed
Needs aligned with CED funding conditions 76%
CED funding conditions changed following a request 17%
CED funding conditions not changed despite a request 4%
Don’t know 3%

Source: Telephone surveys of clients, 2016.

Finding no. 12: The QEDP indirect assistance intervention through intermediary group services adequately met the needs of the businesses assisted, with a satisfaction rate of more than 82% for support services in connection with trade missions and consulting services.

The services deemed most useful by the clients of intermediary groups are:

Assistance provided to Intermediary groups (IGs) is justified. According to the respondents, the value added of IGs lies in their expertise and the diversity of their services, which are targeting the specific needs of local businesses, apart from financial assistance. The cost of the activity or service provided by IGs appears to have little effect on the client’s choice.

3.5. Degree of overlap and complementarity between the QEDP and other comparable programs

The measurement of this indicator is based on two main principles: (1) the experience of CED stakeholders and partners, and (2) telephone survey data.

Finding no. 13: Overlap between CED interventions and the interventions of funders were noted by clients under the Commercialization and exports and Promotion of regional assets components. On the ground, however, there is little overlap between the QEDP and other programs because financial partners collaborate to share costs when funding the same project. Moreover, the rules on stacking government assistance to which public funders are subject limit overlap risk.

With respect to commercialization, for example, CED and BDC have programs with similar objectives, that is, business expansion on international markets. Other funders provide financial assistance for activities similar to those authorized under the QEDP Commercialization and exports component: (i) CanExport from Global Affairs Canada funds new market development; (ii) MESI funds exports, excluding travel; and (iii) Investissement Québec offers the Financement Unique program.

For the Promotion of regional assets component, the Office de tourisme du Québec funds promotion outside Quebec, just like CED.

During interviews, advisors said they had reached informal agreements with other funders over the years in order to split funding for activities linked to a project, since the same funders often contribute to projects funded by CED. Figure 3.5 shows the proportion of projects funded by CED jointly with other financial partners.

According to the advisors interviewed, the presence of other funders is desirable because it has a leverage effect on various sources of funding and allows risks to be shared between partners.

EMoreover, the rules on stacking government assistance to which public funders are subject limit overlap risk. The rate of assistance under the QEDP varies depending on many parameters, such as project type (for example, commercial or non commercial), cost, client type (for example, NPO or business) and MRC (for example, according to the EDI).

Finding no. 14: The majority of operating SMEs and NPOs believe that the interest rates offered by CED (83%), as well as the repayment terms of CED (78%), are more advantageous compared to their other major lender. Only 36% of all recipients indicated CED's administrative requirements as being more beneficial than those of their other major funder. In addition, the multiplicity of donors, each with their own criteria and financing arrangements, is perceived by some beneficiaries to be a source of confusion and could create an administrative burden for beneficiaries.

Figure 3.6 shows that the vast majority of operating SMEs and NPOs to which the situation applies feel that the interest rates offered by CED (83%), as well as CED’s repayment terms (78%) are more advantageous compared to their other primary funder.

Regarding the other four aspects surveyed, 46% feel the level of funding provided by CED to be more advantageous than that of their other primary funder, and one quarter (24%) instead feel this aspect is less advantageous, mainly due to the level of funding and the administrative requirements.

Based on the experience of CED officials, the degree of overlap between QEDP components and the programs of non government funders is low, since CED differs in the following areas:

However, CED stakeholders pointed out that the multiplicity of funders, each with different criteria and funding arrangements, may be perceived by some recipients to be confusing and may create an administrative burden for recipients.

Figure 3.5
Proportion of projects funded by CED jointly with other financial partners (N=713)
description follows

Source: Hermès

Description of figure 3.5
Figure 3.5. Proportion of projects funded by CED jointly with other financial partners
Stakeholder (N = 713) Proportion of projects funded by CED jointly with other financial partners
Client 81 %
Provincial 72,5 %
Private investment 42 %
Federal (other than DEC) 30 %
Financial institution 23 %
Other organizations 18,5 %

Source : DEC information system, Hermès.

Figure 3.6
Comparison between the financial plan offered by CED and those of the other main funder
description follows

Source: Telephone survey of clients, 2006

Description of figure 3.6
Figure 3.6. Comparison between the financial plan offered by CED and those of the other main funder
Characteristics of DEC’s financial aids compare to the other main funder Refusal More advantageous Neither more nor less advantageous Less advantageous
Interest rates (n=632) 3 % 83 % 8 % 6 %
Repayment terms : repayment moratorium and deferment (n=632) 2 % 78 % 12 % 8 %
Level of funding allocated (n=977) 1 % 46 % 29 % 24 %
For assistance to be repayable, non-repayable or a grant (n=977) 2 % 45 % 44 % 9 %
Administrative requirement (n=977) 2 % 36 % 45 % 18 %

Source: Telephone survey of clients, 2016.

3.6. Degree of coordination between stakeholders

This indicator was evaluated based on the experience of clients and CED stakeholders.

Finding no. 15: Coordination between stakeholders was recognized by proponents to obtain funding faster. Although CED collaborates with other partners for better coordination, progress still needs to be made, according to the stakeholders interviewed.

Of the 997 proponents interviewed, 51% saw signs of coordination between CED and other funders. Of this proportion, 81% believe that these exchanges led to obtaining funding faster.

This coordination made the following possible:

Initiatives intended to improve this coordination were undertaken at a number of levels:

3.7. Degree of alignment between government priorities and QEDP objectives

This indicator was evaluated based on a correspondence analysis between official documents outlining government priorities for the 2012–16 period and documents defining the objectives of QEDP components. The following findings flow from this analysis:

Finding no. 16: QEDP interventions are aligned with government priorities, and the objectives of the various QEDP components are in line with the statements of official documents for the period under review.

Table 3.6 shows the alignment between QEDP intervention components and the various departmental and government priorities.

The entrepreneurship support and business development objectives of the Business Development pillar broadly reflect the government priorities set out in various framework documents. For example:

The regional engagement and community investment objectives of the Regional Economic Development pillar broadly reflect the government priorities set out in various framework documents. For example:

Table 3.6
Alignment of QEDP intervention components with government priorities
QEDP component Budget Speech Speech from the Throne Minister’s Mandate Letter
Business creation and startup 2013 and 2016 n/a 2015
Business succession and transfer n/a n/a n/a
Productivity and expansion 2013 to 2016 2013 2015
Innovation and technology transfer 2013 and 2016 2013 2015
Commercialization and exports 2014 and 2016 2013 and 2015 2015
Network structuring (e.g., clusters) 2014 and 2016 n/a 2015
Development strategies (e.g., development and mobilization plans) 2014 and 2016 n/a 2015
Community economic facilities (e.g., economic infrastructure) 2014 to 2016 n/a 2015
Promotion of regional assets (e.g., tourism) 2016 n/a 2015

Source: Internal compilation.

3.8. Degree of alignment between departmental priorities and QEDP implementation

This indicator was evaluated based on a correspondence analysis between reports to Parliament outlining departmental priorities and regional intervention strategies (RISs).

Finding no. 17: The scope of the assistance allocated under the QEDP components is generally consistent with departmental priorities, with a few subtleties observed for the Promotion of regional assets component.

Based on the QEDP components, Table 3.7 shows the connection between CED priorities,Footnote 16 the number of business offices that gave this component priority in their RIS, the number of projects and the assistance approved. The various observations arising from the interpretation of this table follow.

Table 3.7
Number of projects and assistance approved by intervention component (excluding temporary initiatives)
Pillars and intervention components CED priorities 2012–16 Business office having given it priority Number of approved projects Total assistance approved
MS %
Business Development 1,360 517.5 71.3
Business creation and startup Yes 12 203 76.7 10.5
Business succession and transfer n/a n/a 2 0.2 0.03
Productivity and expansion Yes 12 753 266.2 36.7
Innovation and technology transfer Yes 12 119 103.3 14.2
Commercialization and exports Yes 11 249 56.9 7.8
Network structuring n/a n/a 34 14.1 1.9
Regional Economic Development 167 208.3 28.7
Development strategies n/a 3 13 1.7 0.2
Community economic facilities Yes
2015-2016
n/a 13 21.1 2.9
Promotion of regional assets n/a 7 141 185.5 25.5
Total n/a n/a 1,527 725.8 100

Source: Internal compilation.

4. Effectiveness

Questions explored and underlying indicators

What is the degree of usefulness and effectiveness of QEDP performance measurement?

  • 4.1. Degree of effectiveness
  • 4.2. Degree of usefulness

To what extent has the QEDP been effective in achieving the expected results?

  • 4.3. Results achieved in Business Development
  • 4.4. Results achieved in Regional Economic Development
  • 4.5. Results of temporary initiatives supported through the QEDP core budget

What is the leverage effect and what are the other impacts of the QEDP?

  • 4.6. Leverage effect and other impacts of the QEDP

Highlights

  • The application of performance measurement is in keeping with requirements: Performance measurement for CED-supported projects is in keeping with the QEDP performance measurement strategy. A CED analysis reveals some concerns regarding the inaccuracy of the indicators and difficulty in attributing results to the activities carried out.
  • The performance measurement has limited scope for decision making due to the difficulty of attributing the results achieved to the QEDP (such as, attraction of foreign investment) and the time it takes for the results to materialize. These limitations are inherent in regional development programs.
  • The results achieved in Business Development are positive overall: For the period from 2012 to 2016, the survival rate of supported businesses was more than 82%, and the percentage of businesses that maintained or increased their turnover was 61% on average.
  • The results achieved in Regional Economic Development are positive overall: For the period from 2012 to 2016, the projects supported 19 communities and the investments made exceeded initial regional planning and community economic facilities objectives. Also, CED support helped fund 208 canvassing projects for a total of $40.1 million for attracting foreign tourists and investments.
  • The results of the four temporary initiatives funded through the QEDP core budget are positive overall : The Strategic Initiative to Combat the Spruce Budworm Outbreak helped control the infestation in all affected communities. The assistance provided under the Initiative to Rehabilitate Water Crossings and as part of the Bellechasse Pipeline project is progressing according to plan. Through the Local Investment Initiative (LII), 257 infrastructure improvement and community facility projects were completed in 12 communities.
  • The leverage effect generated by the QEDP is $4.91: Every dollar invested through the QEDP generated a direct investment of $4.91 from proponents and other funders, higher than that of programs preceding the QEDP ($3.93).
  • Implementing the QEDP had other positive impacts: Under Business Development, the other impacts include the establishing of new partnerships, the adopting of new business practices and access to new business opportunities in Quebec and abroad. Under Regional Economic Development, they include: community enhancement, increased involvement by other stakeholders and, in some cases, development of off season tourist activities.

4.1. Degree of effectiveness of QEDP performance measurement

The effectiveness of performance measurement was evaluated based on a review of framework documents and an analysis of documents prepared during the attempted update of the QEDP Performance Measurement Strategy in 2012. The following finding was made:

Finding no. 18: Although performance measurement is applied in keeping with requirements, its design could be improved to better reflect the impacts of CED funded projects.

Project performance data are generally compiled as instructed by business office advisors. However, a diagnosis of the state of performance measurement in 2015 suggests the following changes to improve reliability:

For the Business Development pillar:

For the Economic Development pillar:

4.2. Degree of usefulness of QEDP performance measurement

The usefulness of performance measurement was evaluated through interviews with CED stakeholders. The following finding was made:

Finding no. 19: Performance measurement, which was simplified in 2012, is deemed satisfactory and useful by the stakeholders interviewed. However, its scope remains limited for three reasons: (1) difficulty attributing the documented results to the QEDP, (2) difficulty setting targets, and (3) deadlines that pass before the expected results materialize.

The performance measurement simplified in 2012 is satisfactory according to CED’s internal stakeholders. It is deemed useful, particularly for the following activities:

Its usefulness is nonetheless reduced for decision making due to factors intrinsic to regional economic development, as follows:

CED stakeholders identified improvements to both the reliability and the usefulness of performance measurement results, including the following:

4.3. Results achieved in Business Development

The achievement of Business Development targets was evaluated using: (1) data from the CED project information system (Hermès), (2) a comparison between the performance of businesses supported by CED and that of a group of comparable businesses that did not receive CED funding, and (3) impacts observed by stakeholders and project proponents.

Table 4.1 summarizes the results of this pillar. It should be noted that the data on results covers the 2012–16 period, whereas the targets are for the 2012–17 period.

Finding no. 20: With respect to direct assistance to businesses, the targets set for growth in turnover and businesses that increased their international sales were exceeded for the period from 2012–13 to 2015–16. In addition, 38 businesses started up during this period.

Clients generally attribute these gains to spinoffs from initial investments, namely new contracts signed, new markets conquered, improved business models, access to new distribution channels, innovation and improved staff skills.

According to the stakeholders interviewed, the targets are in the process of being achieved given the annual progression in results observed since 2013. In addition, the expected impact of CED interventions can take time to materialize.

Finding no. 21: With respect to interventions conducted through intermediary groups, results in turnover and productivity growth are on the way to achieving the targets set for 2017.

When it set the targets, CED did not have historical data on intermediary group performance, which partly explains the discrepancy between the targets and the results achieved.

Moreover, the stakeholders interviewed believe that the results are in the process of being achieved given the progress observed since 2013 and the presence of certain proven success factors, such as the following:

Table 4.1
Business Development targets and results by focus area (2012–16)
Focus area Number of businesses
Direct assistance
Number of businesses
Indirect assistance
Targets Results Assessment Targets Results Assessment
Business performance Businesses supported 965 800 Satisfactory with improvements 31,200 21,345 Action required
Businesses that increased their turnover 212 225 Satisfactory 285 262 Satisfactory with improvements
Businesses that increased their profit margin 247 282 Satisfactory 43 42 Satisfactory with improvements
Businesses that increased their international sales 42 62 Satisfactory 125 118 Satisfactory with improvements
Survival rate of supported businesses 95 % 99 % Satisfactory n/d n/d n/d
Entrepreneurship support Businesses supported n/a* 153 n/a n/a 1 153 n/a
Businesses started up 59 38 n/a n/a 22 n/a
Survival rates 90 % 100 % Satisfactory n/a n/a n/a

* Targets were not set during the PMS update because the two focuses were supposed to be combined.

Source: Compilation of performance measurement data and results from intermediary group surveys.

Finding no. 22: Interventions in Business Development under the QEDP led to the creation of 59 businesses, supported directly or through intermediary groups. In addition, the business survival rate for all the components combined is close to 99%, and 100% for business startups.

This finding is consistent with the results of the study carried out by Statistics Canada on the performance of CED supported businesses presented in Box 1. This study shows that companies assisted by CED have a higher survival rate than the group of comparable businesses.

Finding no. 23: The results are positive in all components of the QEDP, with the exception of the Commercialization and Exports component, where the proportion of recipients who have maintained or increased their sales, income and profit margin is below target.

According to the stakeholders interviewed, the results of the Commercialization and export component could be attributable to the availability of information, since international sales is not differentiated in the financial statements of the companies.

Box 1. Statistics Canada Study

A comparative analysis conducted by Statistics Canada between the economic performance of businesses that received financial assistance from CED and that of a group of comparable businesses that did not receive such assistance revealed the following:

  • 88% of clients of CED assistance, compared to 84% of comparable businesses, are still operating five years after obtaining financial assistance.
  • Turnover growth reported by clients of CED assistance is slightly weaker than that of comparable businesses, with a 1.2% difference.
  • After 10 years, 69% of businesses supported by CED are still in operation, compared to 61% of the group of comparable businesses.

4.4. Results achieved in Regional Economic Development

The achievement of Regional Economic Development targets was evaluated using: (1) performance indicators for CED interventions shown in Table 4.2 and (2) impacts observed by stakeholders and project proponents.

Finding no. 24: With respect to Regional engagement, interventions made it possible to generate investments above targets, meaning that 19 communities were supported in their development efforts.

These observations are illustrated in Figure 4.1. The investments made under this focus area enabled the 19 communities supported to accomplish the following:

Moreover, other positive impacts from regional engagement activities were highlighted by the clients interviewed, including the creation of a leverage effect and increased cooperation between partners.

Finding no. 25: With respect to the Regional investment focus area, the interventions supported the construction and modernization of community facilities in nine communities. In addition, they helped fund 208 startup projects worth a total of $40.1 million to attract tourists and foreign investment.

As stated in the limitations, the current methodology does not make it possible to measure all the impacts expected from community economic facilities projects. This finding was highlighted by the stakeholders asked about the nature and scope of the impacts of community economic facilities projects.

The success factors they identified are the involvement and cooperation of all levels of government. For risk factors, the various funders including CED identified: the experience of local proponents in planning large scale projects and staying within the established budgets and time frames.

Finding no. 26: Aside from increased tourist traffic, greater visibility of activities put on by supported organizations is another impact of projects funded through the Promotion of regional assets component. These impacts were observed by 63% of the clients interviewed.

Figure 4.2 illustrates the impacts of projects and investments in connection with the promotion of regional assets, where the clients interviewed observed: (1) increased tourist traffic, and (2) other impacts, such as job creation and the region’s visibility with Canadian and foreign tourists.

The interviews conducted with Montréal International (Box 2) and Québec Maritime (Box 3) reveal the challenges of promotion adapted to the assets of various Quebec regions.

Table 4.2
Summary of Regional Economic Development results
Result targets for
2012–13 to 2016–17
Results achieved in
2012–13 to 2015–16
Assessment of achievement of targets
Final outcome
Value of investment generated in supported regions that completed implementation of their projects $9M $21.9M Satisfactory
Number of supported communities that completed their engagement or economic development planning initiatives 20 19 Satisfactory with improvements
Value of total investment generated in supported regions that completed implementation of their community economic facilities projects $29M $40.1M Satisfactory
Value of spending by tourists from outside Quebec attracted to the supported regions $17B $15B Satisfactory with improvements
Value of foreign direct investment maintained in or attracted to supported regions $6B $5.6B Satisfactory with improvements
Intermediate outcome
Number of supported communities carrying out engagement or economic development planning initiatives (DA) 25 19 Action required
Number of supported communities modernizing or acquiring community economic facilities (DA) 10 9 Satisfactory with improvements
Number of tourists from outside Quebec attracted to the regions 30M 25.7M Satisfactory with improvements
Immediate outcome
Number of communities engaged in a mobilization process Not available 19 N/A
Number of community economic facilities being developed or modernized (DA) 12 12 Satisfactory
Number of startup projects aimed at promoting the advantages of tourism Not available 199 N/A
Number of startup projects seeking to attract international organizations or foreign companies Not available 9 N/A

Source: Compilation of performance measurement data.

Figure 4.1
Impacts of regional engagement projects
description follows

Source: Telephone survey of clients, 2016.

Description of figure 4.1
Figure 4.1. Impacts of regional engagement projects
Number of valid respondents =21 Agree Disagree Not applicable
Plan economic development activities and projects 91% 4% 4%
Implement growth generating projects 87% 4% 9%
Have key stakeholders participate 87 % 4 % 9 %
Make funding easier to find 83% 9% 9%
Canvass for opportunities 78% 4% 13%

Source: Telephone survey of clients, 2016.

Figure 4.2
Spinoffs from project implementation in communities
description follows

Source: Telephone survey of clients, 2016.

Description of figure 4.2
Figure 4.2. Spinoffs from project implementation in communities
Number of valid respondents =92 Percentage of respondents
Increased of visitors 35%
Better visibility for the organization 28%
Other (employment, new businesses, profits and renovation) 18%
Media coverage 7%
International visibility 7%
No spinoffs 2%
Don’t know 3%

Source: Telephone survey of clients, 2016.

Box 2. Montréal International

Achievements: Montréal International’s promotional activities helped attract international companies to Montreal and to forge strategic ties between local and non local businesses, thus leading to the retention and creation of 3,240 jobs in 2016.

Success factors identified by the proponent: Compared to other major North American cities, Montreal has the following advantages:

  • A wider range of funding opportunities,
  • Typically lower implementation costs,
  • Relatively more flexible regulations for sectors with high value added.

Risk factors: At the prospecting stage, foreign companies generally look at access to funding. A letter of intent or recommendation from government funders can be a solution to encourage foreign investors to choose Quebec.

Source: Interviews

Box 3. Québec Maritime

Achievements: The organization’s promotion activities helped increase tourist traffic, particularly off season, thereby contributing to job creation.

Success factors identified by the proponent: The success of Québec Maritime activities depends on a strategy to promote the region’s tourism products in an integrated way, which ultimately led to greater visibility for the entire region.

Risk factors: Limited transportation offerings in remote regions and the poor fit between government program criteria and the needs of organizations such as Québec Maritime are some of the risk factors identified by the stakeholders interviewed.

Source: Interviews

4.5. Results of the four temporary initiatives supported through the QEDP core budget

  1. Strategic Initiative to Combat the Spruce Budworm Outbreak in Quebec

    From 2014 to 2018, CED allocated $6 million in non repayable contributions to implement the Strategic Initiative to Combat the Spruce Budworm Outbreak in Quebec. The spruce budworm is an insect that ravages coniferous forests, causing major business losses for the forestry industry. The purpose of this initiative is to ensure the viability of this industry and to protect the jobs of workers that depend on it. CED deals with the Société de protection des forêts contre les insectes et maladies (SOPFIM), which carries out activities to combat the spruce budworm, such as spraying with insecticides.

    As at March 31, 2016, a total of 21,450 hectares were sprayed as part of this project. As a result of this initiative, the SOPFIM optimized its intervention strategies by making significant efficiency gains (spraying double the area at the same cost). These gains maximize SOPFIM interventions, which helps maintain forest potential, jobs and the economic vitality of communities that depend on it.
  2. Bellechasse MRC Pipeline

    In 2015, the Treasury Board allocated $17.5 million to CED to fund a pipeline between the municipalities of Lévis and Sainte Claire in the Bellechasse MRC. This project contributed to the achievement of a government priority to create jobs and stimulate economic growth.

    Many manufacturing industries are located in this MRC, and access to natural gas, a source of low cost energy, represents significant savings for businesses. In March 2017, work to install the main pipe was finished and the new section was commissioned. The target of 90 connections by 2020 had already been exceeded, as 97 businesses were connected by March 2017. More connections were also expected.
  3. Initiative to Rehabilitate Water Crossings on Wildlife and Multi resource Roads

    This initiative has a budget envelope of $9.6 million over three years (2014–17).

    CED funded projects that seek to encourage the rehabilitation of water crossings damaged by aging and adverse weather, and to improve, rehabilitate and maintain these wildlife and multi resource roads.

    In 2015–16, $4.1 million was spent as part of this Canada–Quebec agreement to fund projects.
  4. Local Investment Initiative

    CED funded projects to support local investment in all Quebec regions by encouraging the renovation, improvement or extension of existing public and non commercial community halls.

    In 2015–16, CED spent $5.1 million on 137 projects.

4.6. Leverage effect and other impacts

The QEDP leverage effect was calculated using the ratio between the number of dollars invested by CED and the number of dollars invested from other sources. The others were identified during a telephone survey of clients.

The main findings are discussed below.

Finding no. 27: For the period from 2012-2013 to 2015-2016, each QEDP dollar generated a direct investment of $4.19 from proponents and other financial partners, for a total of $4.7 billion. The leverage effect of Regional Economic Development assistance is relatively higher than that of Business Development.

Table 4.3 shows the leverage effect by dollar invested in the various QEDP components. The Business creation and startup component generates the largest leverage effect, followed by Promotion of regional assets and Network structuring.

The overall leverage effect is higher than the one recorded for CED programs prior to 2012, which was $3.93 per dollar invested. The stakeholders interviewed said that this difference is mostly due to contributions from other financial partners during the evaluation period.

Finding no. 28: Other impacts are few and far between, and are generally positive. In Business Development, they include the creation of new partnerships, the learning of new practices and access to new business opportunities. In Regional Economic Development, they include community enhancement, increased involvement by other stakeholders and, in some cases, development of off season activities.

Given the maturity of the projects under review, it is difficult to identify additional impacts. The few additional impacts identified are generally positive.Footnote 18

In Business Development, these positive implications include:

In Regional Economic Development, they include the following:

Figure 4.3
Other impacts in percentage of clients interviewed
description follows

Source: Telephone survey of clients, 2016

Description of figure 4.3
Figure 4.3. Other impacts in percentage of clients interviewed
Other impacts Business development component of program Regional economic development component of the program
No direct positive or negative implications 47.7 % 57.7 %
Indirect positive implication 17.6 % 17.7 %
Indirect negative implication 6.6 % 2.1 %
Not valid 1.9 % 1.3 %
N valid = 1,267 26.1 % 21.2 %

Source: Telephone survey of clients, 2016.

Table 4.3
Leverage effect by intervention component and by client type
Pillars and intervention components Leverage effect ($)
NPO SME Overall
Business Development 2.5 5.8 4.84
Business creation and startup 3.5 13.7 11.61
Business succession and transfer 2.4 n/a 2.4
Productivity and expansion 1.9 4.3 3.94
Innovation and technology transfer 1.3 5.3 2.63
Commercialization and exports 4.6 2.4 3.09
Network structuring 6.6 n/a 6.6
Regional Economic Development 6.77 n/a 6.77
Development strategies 1.71 n/a 1.71
Community economic facilities 1.36 n/a 1.36
Promotion of regional assets 7.59 n/a 7.59
QEDP leverage effect n/a n/a 4.91

Source: Hermès

5. Efficiency

Questions explored and underlying indicators

Were there any problems and facilitating factors during the various phases of the QEDP life cycle?

  • 5.1. Problems and facilitating factors during QEDP design and implementation
  • 5.2. Impacts of CED organizational changes on QEDP implementation

How can QEDP efficiency be improved?

  • 5.3. Work organization
  • 5.4. Operating costs
  • 5.5. Client relationship
  • 5.6. Time dedicated to project management
  • 5.7. Project risk level
  • 5.8. Service delivery method

Highlights

  • No major problems were identified during the QEDP design and implementation phases in 2012: Since the program’s implementation, build-up of the various departmental guidelines, directives and strategies have made program delivery more complex, according to some advisors interviewed.
  • Organizational changes within CED, coupled with smaller travel budgets, affected the efficiency of QEDP implementation between 2012-2013 and 2015-2016: The main impact identified is more limited availability of business office advisors for carrying out the QEDP in the field. However, CED spent its budget allocation in grants and contributions during that period.
  • The organization of the work is efficient overall. The business processes and work tools could benefit from improvements, according to some CED stakeholders.
  • The program is delivered by optimizing financial resources: Operating costs account for 17,7 % of expenses incurred for the period under review. These costs include CED employee wages and expenses associated with QEDP implementation.
  • 92% of QEDP clients interviewed are satisfied with their relationship with CED. The less satisfied clients (8%) want simpler administrative processes, financial assistance eligibility criteria adapted to their needs and more support with interpreting the documentation
  • The type of assistance used has an impact on the time spent on each step of project management: The time spent on grant management is shorter than the time spent managing non-refundable contribution. The time spent on the management of non-refundable contributions is also shorter than that for the management of refundable contributions. Thus, efficiencies could be achieved by using non-refundable grants and contributions, when possible.
  • Applying the risk profile resulted in increased efficiency by altering the time spent on managing projects based on risks. That efficiency increase is 28 hours per project.
  • The service delivery method meets most of the needs of the clients interviewed: Some clients pointed out the difficulty accessing QEDP funding in some MRCs with low economic potential, which would explain the few projects submitted.

5.1. Problems and facilitating factors during QEDP design and implementation

The problems and facilitating factors were analyzed based on the program’s framework documents and interviews with CED internal stakeholders. The following findings were made:

Finding no. 29: The stakeholders interviewed did not indicate any major problems. From their point of view, the establishment of a project office dedicated to the QEDP, the creation of a multi sectoral committee, the allocation of resources specific to the approach and the involvement of senior management were factors that facilitated program design.

A project team was mandated to renew the strategic directions, the program activity architecture (PAA), the QEDP design, and implementation tool development and set up.

In addition, a multi sectoral committee was created and held productive consultations with all CED sectors.

The project office with its dedicated resources and involvement by senior management, which began the work early on, are factors that contributed to the program’s successful design and implementation.

Although no major problems were identified, improvements and examples of good practices were gathered during the post mortem survey conducted of all CED sectors that had taken part in this exercise:

Finding no. 30: QEDP implementation was smooth, particularly owing to more functional tools and adapted training. The many departmental directives and guidelines added since program implementation made the program more complicated to roll out.

According to internal stakeholders, QEDP implementation was smooth, particularly because it provides continuity from previous programs. Functional tools and training adapted to the needs of CED business office advisors were brought up; they would have preferred that the training be held earlier.

On April 1, 2012, the CED En route Initiative was launched to coordinate QEDP implementation. This initiative also integrated the results of the CED modernization exercise, that is, streamlined processes, simplified reporting and redesigned regional strategies for business offices.

The business office advisors interviewed stated that, since the QEDP’s launch, the juxtaposition of numerous ministerial guidelines, directives, processes and strategiesFootnote 19 has created confusion in program implementation.

The quality audit carried out in 2013–14 also made this finding and recommended streamlining and/or limiting the number of directives.

5.2. Impacts of CED organizational changes on QEDP implementation

The interviews conducted with CED stakeholders gave rise to the following finding:

Finding no. 31: The organizational changes in 2012 following the implementation of the Deficit Reduction Action Plan and the end of temporary programs, coupled with travel budget cuts, reduced CED’s visibility in the regions and limited collaborations with partners. However, CED spent its budget allocation in grants and contributions during that period.

The launch of the QEDP in April 2012 coincided with organizational changes, following the implementation of the Deficit Reduction Action Plan and the end of two temporary programs (Community Adjustment Fund and Recreational Infrastructure Canada Program). The main changes were:

In anticipation of these changes, CED undertook an extensive modernization process to make the Agency more efficient and provide better service to the public. As a result, the guiding principles of the QEDP were modernization, adaptation and simplification.

During interviews, CED representatives mentioned the following impacts:

Also, during the period studied, only $258,965 was allowed to lapse in 2013-2014, and CED was able to spend the QEDP budget allocation in grants and contributions.

Table 5.1
Travel expenses from 2007-08 to 2015-16
Years Travel expenses (thousands of $)*
2007-2008 1,308.4
2008-2009 1,372
2009-2010 1,145
2010-2011 1,003.3
2011-2012 783.2
2012-2013 535.2
2013-2014 525.1
2014-2015 574.8
2015-2016 683.8

Note: The amounts shown are not in constant dollars.

Source : CED Financial statements.

5.3. Work organization

The findings below were made following interviews with CED stakeholders regarding work efficiency:

Finding no. 32: Avenues for improvement were suggested for optimizing CED business processes. The efficiencies resulting from those suggestions are still to be documented.

CED stakeholders interviewed about efficient organization of work felt that CED business processes could be optimized, and they suggested the following avenues:

The efficiencies resulting from the suggested avenues for improvement still have to be documented.

Finding no. 33: The work tools for QEDP implementation are deemed appropriate according to internal stakeholders. However, they want them to be better integrated.

The work tools for QEDP implementation are deemed appropriate by their users. These tools would benefit from being better integrated.

Interest in integrating available tools was already expressed as part of the Blueprint 2020 initiative launched in 2013 to modernize the public service.

Finding no. 34: The advisors responsible for QEDP delivery feel they received the training they need to deliver the QEDP. They would also like access to sector based specialization.

According to the 2014 Public Service Employee Survey, 80% of employees in the operations sector felt they received the training they needed to do their work. During interviews, the advisors responsible for QEDP delivery said they had the appropriate knowledge and skills to implement the program.

Some advisors also mentioned wanting to specialize in sectoral analysis because, for them, having a thorough understanding of a sector before going ahead with its development gives them a better tactical vision and allows them to better support proponents in developing their projects, thereby increasing the value added of “non financial” assistance from CED.

Finding no. 35: The advisors responsible for delivering the QEDP regularly need internal and external information sources to analyze and monitor supported projects. To that end, they want more formal support networks to be set up in order to help proponents more effectively.

In addition to the tools already available to them, the advisors need internal and external support networks to analyze projects requiring specific expertise (such as an innovation project in a complex scientific field) and to have a clear understanding of the economic issues specific to each territory.

On the one hand, they want a mechanism to be developed for easier access to the following resources: (i) directory of partners with varied expertise in connection with intervention priorities; and (ii) policy sector products on issues specific to their territory.

On the other hand, they want agreements with partners to be revised in order to produce sectoral notices so that they do not take longer to process applications.

Finding no. 36: To deliver temporary programs without increasing its operating costs, CED had to reduce its employees’ presence on the ground.

The interviews conducted with CED stakeholders revealed that temporary program implementation increases advisors’ workload, thereby reducing their presence on the ground to implement the QEDP.

5.4. Operating costs

Finding no. 37: Operating costs accounted for 17,7 % of expenses incurred for the 2012-2013 to 2015-2016 period.

As illustrated in Table 5.1, this proportion is relatively higher than the average for the five years preceding QEDP implementation, when operating costs represented 16.8% of incurred expenses.

During this period, more transfer payments were made with essentially the same operating budget due to, among other things, the implementation of temporary programs requiring less project monitoring. One such program was the Community Adjustment Fund (CAF) in 2010–11.

Figure 5.1
Change in the proportion of operating costs and transfer payments, 2012-13 to 2015-16
description follows

Source: Public Accounts of Canada, 2006 to 2017

Description of figure 5.1
Figure 5.1. Change in the proportion of operating costs and transfer payments, 2012-13 to 2015-16
Years Transfer of payment (millions of $) Operating costs (millions of $)
2005-2006 282.9 51.2
2006-2007 316.1 48.7
2007-2008 285.8 50.2
2008-2009 242.5 50.7
2009-2010 341.9 56.0
2010-2011 424.1 55.8
2011-2012 252.6 53.2
2012-2013 247.5 48.9
2013-2014 225.9 43.4
2014-2015 212.2 40.7
2015-2016 218.5 40.7

Source: Public Account of Canada.

Figure 5.2
Client satisfaction with various aspects of the funding process
description follows

Source: Telephone survey of clients, 2016.

Description of figure 5.2
Figure 5.2. Client satisfaction with various aspects of the funding process
Assessed Item (N = 1328) Totally agree Somewhat agree Somewhat disagree Totally disagree
Formalities simple and easy to complete 60 % 32 % 5 % 1 %
Reasonable response times for assistance applications 67 % 26 % 4 % 2 %
Administrative documents easy to understand 68 % 26 % 3 % 1 %
Programs and eligibility criteria easy to understand 63 % 31 % 5 % 1 %
Reasonable processing times for claims 78 % 18 % 2 % 1 %
Reasonable response times for enquiries 83 % 14 % 2 % 0 %
Service easy to access through various channels 82 % 15 % 1 % 0 %
Information and advice meet needs 85 % 16 % 0 % 0 %

Source: Telephone survey of clients, 2016.

5.5. Client relationship

The following findings stem from the telephone survey conducted with clients:

Finding no. 38: All evaluated aspects of the funding process received a satisfaction rate above 92%. Dissatisfied clients wanted mainly simpler administrative processes, eligibility criteria that take into account the diversity of contexts and support for interpreting documents.

Figure 5.2 illustrates clients’ satisfaction with various aspects of the funding process, with satisfaction rates above 92% for both information provided regarding terms and conditions and claim processing times.

The proportion of dissatisfied clients is less than 6% for the various aspects of the funding process. Clients were least satisfied with:

Dissatisfied clients wanted mainly simpler administrative processes, eligibility criteria that take into account the diversity of contexts and support for interpreting documents.

Additional interviews conducted with local stakeholders made it possible to identify factors contributing to greater client satisfaction, such as continued relationships over the years, bilingual capacity, advisor availability and flexibility with constraints linked to clients’ business lines. According to these local stakeholders, clients would like advisors to have a better knowledge of their business line (for example, better tourism expertise) and would like the performance reports requested by various funders to be aligned.

As for intermediary group clients, a high satisfaction rate (70% were completely satisfied) was observed in connection with their services, particularly for networking, introduction to new markets, support for innovation and business planning.Footnote 22.

Lastly, even though the satisfaction rate of QEDP clients is high overall, they vary slightly from one QEDP component to the next. For example, 48% of clients in the Commercialization and exports component were completely satisfied that the program and eligibility criteria were easy to understand, compared to 63% for all clients.

Finding no. 39: The response times for financial assistance applications were met in more than 80% of projects since QEDP implementation.

This finding is based on service standard data (response time between 35 and 65 days, maximum). Exceptions to this trend are the following:

Moreover, the telephone survey revealed that 94% of proponents found the time it took to respond to their financial assistance applications reasonable compared to other funders. For some dissatisfied proponents, this delayed their project implementation.

5.6. Time dedicated to project management and risk level

The AIS data are collected through sampling, six out of twelve months for each advisor. To estimate the time dedicated to project management, the time in the AIS was doubled. The following findings arise from this data analysis:

Finding no. 40: The time dedicated to managing grants is shorter than what is dedicated to managing non-refundable contributions, at all phases of project management. The time dedicated to managing non-refundable contributions is also shorter than for managing refundable contributions. Therefore, allocating non-refundable grants or fundable contributions for low risk projects with small assistance amounts would improve efficiency.

This finding is based on the AIS data shown in Figure 5.3. For example, the time spent managing grants of less than $15,000 is 31.4 hours, compared to 45.7 hours for non repayable contributions for the same amount. That is close to one third less time.

Grants are usually allocated in a lump sum, do not need to be repaid and are not subject to reporting. Non repayable contributions may require multiple payments and are subject to reporting. Repayable contributions may also require multiple payments and are generally repaid in several installments. Repayable contributions therefore require more processing and monitoring than grants and non repayable contributions.

Finding no. 41: Applying the risk profile alters the time dedicated to project management. That efficiency increase is 28 hours per project.

In 2013, CED introduced a new risk profile for businesses designed to determine the type of monitoring to be carried out and whether or not mitigation measures are required (for example, taking collateral).

It consists in analyzing the following four components:

  1. Preliminary evaluation (repayment risk);
  2. Financial capability;
  3. Operational capacity and business environment;
  4. Client history.

Overall, an analysis of the AIS data shows that the application of the risk profile adjusts the time dedicated to project management. According to the information presented in Figure 5.4, the time dedicated to managing projects deemed to be non risky is 68.8 hours, compared to 97 hours for projects deemed to be risky, that is, 29% less time.

Furthermore, some of the stakeholders interviewed are of the opinion that mitigation measures such as taking collateral or another risk mitigating measure creates an excessive workload for both proponents and advisors.

Figure 5.3
Number of hours dedicated to projects by expenditure band and by funding type between April 2014 and March 2016
description follows

Source: AIS

Description of figure 5.3
Figure 5.3. Number of hours dedicated to projects by expenditure band and by funding type between April 2014 and March 2016
Expense Instalment Application preparation Claim processing Project monitoring
$15,000 and less (grants) 25.7 4.2 1.5
$15,000 and less (non-repayable) 31.9 8.6 5.2
$15,001 to $50,000 (repayable) 55.1 21.4 8
$15,001 to $50,000 (non-repayable) 29.7 12.9 6.7
Over $50,000 (repayable) 57.8 23.2 13.8
Over $50,000 (non-repayable) 40.9 15.6 10.7

Source: CED information system, SIA.

Figure 5.4
Number of hours dedicated to contribution projects by risk level between April 2014 and March 2016
description follows

Source: CED information system, SIA.

Description of figure 5.4
Figure 5.4. Number of hours dedicated to contribution projects by risk level between April 2014 and March 2016
Activities Non-risky Risky
Application preparation 42,9 59,7
Claim processing 16,2 23,2
Project monitoring 9,7 14,1

Source: DEC information system, SIA.

5.7. Project risk level

Finding no. 42: Roughly 62% of projects approved during the period under review are low risk. In addition, CED’s risk taking has remained steady over the years, with an average loss rate of 10% between 2007 and 2016. Internal stakeholders suggest that CED should take more risks for the types of projects that usually fall under the Creation and start-up and Innovation and technology transfer components.

Between 2007 and 2016, CED maintained a fairly steady loss rate. In 2016, this rate was 10% for projects approved under the QEDP, whereas it was 6% and 12% for the two programs that preceded the QEDP. This rate varies from one intervention component to the next, and is higher for Creation and startup.

Asked about risk taking, internal stakeholders stated that CED could take more risks, particularly for Creation and startup and innovation projects.

A review of risk profiles with a dedicated envelope for risky projects and subsequent targets was also mentioned as an improvement.

5.8. Service delivery method

Finding no. 43: For internal stakeholders, the “on demand” QEDP delivery model is suitable. It could, however, be constantly adapted to the clientele and business lines. Advisors could also play a support role, particularly in some MRCs with low economic potential.

According to internal stakeholders, this model is adapted to the reality of businesses. Other approaches that give greater preference to non repayable contributions could, however, be considered for NPOs whose activities have significant indirect economic spinoffs.

Stakeholders indicated that CED could review advisors’ interventions to ensure that advisors are even more present on the ground and can play a support role. This would help reach MRCs with low economic potential where few projects only are funded under the QEDP.

Lastly, some stakeholders stated that the QEDP approach is consistent. However, various QEDP components cover diverse fields with their own particularities, needs and target clientele.

6. Conclusion

The evaluation was conducted in order to determine the extent to which the QEDP remains relevant, reflects the guidelines established to achieve the expected results and has been efficient. The tables below present the conclusions in this regard.

Table 6.1
Conclusion regarding QEDP relevance
Questions Conclusions Results
Did the QEDP meet needs? How have these needs changed over time? The program has met the needs for which it was implemented in 2012: A comparison of the needs underlying the QEDP and current needs in the regions shows that the issues and challenges justifying the program’s continuation remain. Thus, the need to stimulate innovation for small businesses is still present. Satisfactory
The program is vital for most projects: Without CED’s financial assistance through the QEDP, most projects would not have been carried out. Satisfactory
The value of the assistance approved is higher in regions with strong economic potential where the number of financial assistance applications is higher: Although the value of the assistance approved is proportional to the demographic weight of the regions, the assistance amounts and the number of projects are relatively higher in regions with strong economic potential than in those with low economic potential. Action required
Adaptation of interventions to needs
The QEDP delivery model segmented by component does not always meet business and regional needs: The many eligibility criteria specific to each QEDP component make the interventions less agile and, in some cases, can increase the time it takes to deliver them to clients. Action required
Adaptation of interventions to needs
How does the QEDP complement or overlap with other interventions? The program does not overlap with other interventions, but the large number of funders is a challenge for some clients: There is little overlap because funding needs are high and financial partners are bound by assistance stacking rules. Nonetheless, the large number of funders risks creating confusion or even causing an administrative burden for clients. Action required
Improvement of client experience
Coordination between stakeholders is observed and appreciated by clients: Most QEDP clients have noted some coordination between stakeholders on the ground. However, according to the stakeholders interviewed, there is still progress to be made. Action required
Improvement of client experience
How is the QEDP aligned with government priorities? CED’s priorities for the QEDP are aligned with government priorities: During the period under review, departmental priorities were in line with those of the government, and the objectives underpinning the QEDP components were consistent with government priorities set out in official documents. Satisfactory
The amount of the assistance awarded under the QEDP is generally consistent with Departmental priorities with some nuances observed for the Promotion of Regional Assets component, mainly due to projects receiving recurring funding in some regions. Action required
Adaptation of interventions to needs
Table 6.2
Conclusion regarding QEDP effectiveness
Questions Summary of observations Results
What is the degree of usefulness and effectiveness of performance measurement? Application of performance measurement is consistent with the requirements: Although performance measurement for CED supported projects is in keeping with the QEDP performance measurement strategy. A CED analysis reveals some concerns regarding the inaccuracy of indicators and difficulty in attributing results to the activities carried out. Action required
Data and information precision
Performance measurement has a limited scope for decision making: The scope of performance measurement is reduced, in particular, by difficulties in attributing documented results to the QEDP (such as attraction of foreign investment) and setting targets for some activities. In addition, some time passes before the expected outcomes materialize. Action required
Data and information precision
To what extent has the QEDP been effective in achieving the expected results? The results achieved in Business Development are positive overall: For the period from 2012 to 2016, the survival rate of supported businesses was more than 82%, and the percentage of businesses that maintained or increased their turnover was 61% on average. Action required
Data and information precision
The results achieved in Regional Economic Development are positive overall: For the period from 2012 to 2016, the projects supported 19 communities and the investments made exceeded initial regional planning and community economic facilities objectives. Also, CED support helped fund 208 canvassing projects for a total value of $40.1 million for attracting tourists and foreign investments. Action required
Data and information precision
The results of the four temporary initiatives funded through the QEDP core budget are positive overall: The Strategic Initiative to Combat the Spruce Budworm Outbreak helped control the infestation in all affected communities. The assistance provided under the Initiative to Rehabilitate Water Crossings and as part of the Bellechasse Pipeline project is progressing according to plan. Through the Local Investment Initiative (LII), 257 infrastructure improvement and community facility projects were completed in some 10 communities. Satisfactory
What is the leverage effect and what are the other impacts of the QEDP? The leverage effect generated by the QEDP is $4.91: Every dollar invested through the QEDP generated a direct investment of $4.91 from proponents and other funders, higher than that of programs preceding the QEDP ($3.93). 4,91 $
Implementing the QEDP has had other positive impacts: In Business Development, other impacts include the creation of new partnerships, the adopting of new business practices and access to new business opportunities in Quebec and abroad. In Regional Economic Development, they include community enhancement, increased involvement by other stakeholders and, in some cases, development of off season tourist activities. Satisfactory
Table 6.3
Conclusion regarding QEDP efficiency
Questions Summary of observations Results
Were there any problems and facilitating factors during the various phases of the QEDP life cycle? No major problems were identified during the QEDP design and implementation phases in 2012: The build-up of the various Departmental guidelines, directives and strategies has made program delivery more complex, according to some advisors interviewed. Action required
Improvement of client experience
Organizational changes within CED, coupled with smaller travel budgets, have affected the efficiency of QEDP implementation during the 2012-2013 to 2015-2016 period: The main impact noted is the more limited availability of business office advisors to promote the QEDP on the ground. However, CED spent its budget allocation in grants and contributions during that period. Action required
Optimization of program efficiency
How can QEDP efficiency be improved? The organization of work is efficient overall. The business processes and work tools would benefit from improvements, according to some CED stakeholders. Action required
Optimization of program efficiency
The program is delivered by optimizing financial resources: Operating costs account for 17.7 % of expenses incurred for the period under review. These costs include CED employee wages and expenses associated with QEDP implementation. Satisfactory
92 % of QEDP clients are satisfied with their relationship with CED. Less satisfied clients (8%) would like simpler administrative processes, financial eligibility criteria adapted to their needs and more support with interpreting the documentation. Satisfactory
The type of assistance used has an impact on the time spent on each step of project management: The time spent on grant management is shorter than the time spent managing non-refundable contribution. The time spent on the management of non-refundable contributions is also shorter than that for the management of refundable contributions. Thus, efficiencies could be achieved by using non-refundable grants and contributions where possible. Action required
Adaptation of intervention to needs
Applying the risk profile resulted in an increase in efficiency, by altering the time spent on managing projects based on risks. That efficiency increase is 28 hours per project. Action required
Adaptation of intervention to needs
The service delivery method meets most of the needs of the clients interviewed: However, some clients pointed out the difficulty of accessing QEDP funding in some MRCs with low economic potential, which would explain the few projects submitted. Action required
Improvement of client experience

7. Action Plan

The evaluation shows that, overall, the QEDP is relevant, achieves the expected results and is managed efficiently. Based on the various evaluation findings, below are six recommendations pertaining to four themes: (1) availability, quality, and reliability of data and information; (2) improvement of client experience; (3) adaptation of intervention to the needs; and (4) optimization of the program efficiency.

Table 7.1
Excerpts of evaluation findings and recommendations
Excerpts of evaluation findings and recommendations Recommendations
Theme 1: Availability, quality, and reliability of data and information
  • Section 2.5. Evaluation limitations: The performance measurement strategy was updated between April 2015 and March 2016, but considering that the Policy on Results was supposed to be coming out shortly, the recommended changes, including the targets and replacement of some indicators, were not approved. The evaluation is nonetheless based on the available targets.
Recommendation 1: Every five years, CED should establish and approve, in a timely manner, targets for QEDP outcome indicators.
  • Section 2.5. Evaluation limitations: Client satisfaction was evaluated using the database of clients whose projects received financial assistance from CED. Since CED does not document its informal refusals, the evaluation could not cover the perspective of other businesses that did not receive assistance.
Recommendation 2: To better assess client satisfaction and alignment with needs, CED should document its interactions with all project proponents, including those that received an informal refusal from CED.
  • Finding no. 18: Although performance measurement is applied in keeping with requirements, its design could be improved to better reflect the impacts of CED funded projects.
Recommendation 3: CED should ensure that:
  1. Its results chains are consistent with funded activities;
  2. Its performance indicators are always linked to its activities.
Theme 2: Improvement of the client experience
  • Finding no. 14: The majority of operating SMEs and NPOs that the situation applies to believe that the interest rates offered by CED (83%), as well as the repayment terms of CED (78%), are more advantageous compared to their other major lender. Only 36% of all recipients indicated CED's administrative requirements as being more beneficial than those of their other major funder. In addition, the multiplicity of donors, each with their own criteria and financing arrangements, is perceived by some beneficiaries to be a source of confusion and could create an administrative burden for beneficiaries.
  • Finding no. 38: All evaluated aspects of the funding process received a satisfaction rate above 92%. Dissatisfied clients wanted mainly simpler administrative processes, eligibility criteria that take into account the diversity of contexts and support for interpreting documents.
Recommendation 4: To improve the client experience, CED should continue to simplify its administrative processes and work in collaboration with other funders to minimize the collective administrative burden on clients.
Theme 3: Adaptation of interventions to needs
  • Finding no. 10: In order to better align the QEDP with changing business and regional needs, CED stakeholders suggested improvements that include revising the eligibility criteria for the Business succession and transfer and Promotion of regional assets components.
  • Finding no. 34: The advisors responsible for QEDP delivery feel they received the training they need to deliver the QEDP. They would also like access to sector based specialization.
  • Finding no. 43: For internal stakeholders, the “on demand” QEDP delivery model is suitable. It could, however, be always adapted to the clientele and business lines. Advisors could also play a support role, particularly in some MRCs with low economic potential.
  • Finding no. 42: Roughly 62% of projects approved during the period under review are low risk. In addition, CED’s risk taking has remained steady over the years, with an average loss rate of 10% between 2007 and 2016. Internal stakeholders suggest that CED would benefit from taking more risks for the types of projects that usually fall under the Creation and startup and Innovation and technology transfer components.
Recommendation 5: To better understand the needs of businesses and regions in Quebec, CED should ensure that:
  1. Its service delivery model and eligibility criteria for the QEDP are tailored to the specifications of the target clientele;
  2. Its advisors have or can easily access the specialized expertise they need to analyze projects and better support proponents.
Theme 4: Optimization of the program efficiency
  • Finding no. 30: QEDP implementation was smooth, particularly owing to more functional tools and adapted training. The many departmental directives and guidelines added since program implementation made the program more complicated to roll out.
  • Finding no. 32: The business processes are efficient and, avenues for improvement were suggested by the CED stakeholders interviewed. However, the efficiencies resulting from those suggestions are still to be documented.
  • Finding no. 40: Grants take less time to manage than non repayable contributions at all phases of project management. Non repayable contributions also take less time to manage than repayable contributions. Therefore, allocating grants or non repayable contributions for low risk projects with small assistance amounts would improve efficiency.
Recommendation 6: To optimize the efficiency of implementing the QEDP, CED should:
  1. Continue to integrate various departmental guidance tools;
  2. Document the cost effectiveness of using various types of contributions based on risk level and the assistance amounts allocated;
  3. Document the efficiencies achieved further to establishing new business processes.
Table 7.2
Action plan
Recommendations Management Response Timeframe Responsibility Centre
Availability, quality and reliability of data and information
1. Every five years, CED should establish and approve, in a timely manner, targets for QEDP outcome indicators. With the implementation of the Government's new Policy on Results on April 1, 2018, CED has drafted its Performance Information Profile (PIP) for current programs. CED has established indicators for the QEDP program and will develop performance targets for fiscal years 2018-2019 to 2023-2024. Summer - Fall 2018 Policy, Economic Intelligence and Results Branch (PEIRB)
2. To better assess client satisfaction and alignment with needs, CED should document its interactions with all project proponents, including those that received an informal refusal from CED. As part of the development of the new grants and contributions management system, CED will deploy a first module on customer relationship management. This module provides support for interactions with customers, prior to the request for financial assistance. In addition to contributing to the improvement of the service offered to clients, the module will document and optimize the management of data related to customer interactions, including: embryonic requests, informal rejections, referencing to other departments and partners, requests for general information and advice and customer contacts.

This module will be deployed as a pilot in June 2018 to four business offices (BA) prior to the deployment to all BAs in September 2018.
Fall 2018 Client Experience and Business Information Directorate (CEBID)
3. CED should ensure that:
  • 3a. Its results chains are consistent with funded activities;
CED will pay particular attention to identifying the most relevant indicators for funded projects, including standardizing indicators for project categories (e.g. renewals of NPOs, and specific initiatives) and quality assurance at the time of the project review for approval. Continuous Regional Operations Branch (ROB)
  • 3b. Its performance indicators are always linked to its activities.
In accordance with the requirements of the Policy on Results, CED submitted a PIP for the QEDP program to the Treasury Board Secretariat in November 2017. The PIP includes results chains and funded activities as well as performance indicators. CED intends to review its PIP once a semester in conjunction with the biannual review of interventions to ensure that results chains, funded activities and indicators are consistent. Each end of semester for the next five years PEIRB
Improvement of client experience
4. To improve the client experience, CED should continue to simplify its administrative processes and work in collaboration with other funders to minimize the collective administrative burden on clients. In 2018-2019, CED will seize the opportunity of the new Grants and Contributions Management System deployment to review its business processes by modernizing and simplifying ways to improve the customer experience. As part of this revision, external client consultations were conducted to identify areas for improvement in their experience with CED at all stages of a project: from the application receipt to reimbursement of the contribution. These improvements are in addition to those suggested by the employees. Deployment of the new system including the revised processes is scheduled for the summer of 2019. Summer 2019 CEBID
Adaptation of intervention to the needs
5. To better understand the needs of businesses and regions in Quebec, CED should ensure that:
  • 5a. Its service delivery model and eligibility criteria for the QEDP are tailored to the specifications of the target clientele;
CED will continue to ensure, based on its priorities and those of the government, to take into account the needs of businesses in the regions of Quebec in the implementation of its programs and initiatives, and to adapt its delivery model according to the targeted clients and the priority activity sectors.

In 2016, to develop its new 2021 Strategic Plan, CED embarked on a "Dialogue on the Economic Development of Quebec Regions" Engagement Strategy, which attracted more than 1,000 participants from all regions of Quebec. The opinions and ideas expressed allowed CED to improve the implementation of its program in order to meet the needs of Quebec's SMEs and regions by developing, among other things, new components for the QEDP.

CED will continue to implement its engagement strategy by initiating client consultations (e.g. roundtable discussions with regional economic development stakeholders, panel discussions with women) to ensure that its interventions meet the needs of the community by making adjustments to its parameters or criteria, if necessary.

In order to take into account the specific needs of certain under-represented groups, CED has developed in 2017 new intervention parameters targeting Aboriginal people. Analyzes are under way to adjust the intervention parameters to the needs of women entrepreneurs and to stimulate business innovation.

CED will soon be able to use new generic terms developed by the Treasury Board Secretary as part of a pilot project. These new terms and conditions will allow greater flexibility in the delivery of services to Québec businesses and regions. For the QEDP, the new terms and conditions will help create new authorizations, provisions and exceptions through the use, of incentive-based and price-based funding.
Fall 2019 PEIRB
  • 5b. Its advisors have or can easily access the specialized expertise they need to analyze projects and better support proponents.
As part of its integrated resource planning, CED has identified adviser training as a priority to empower employees with skills of the future. Considering that the customers' business model is constantly changing, employees must always be informed and acquire expertise, especially in new technologies. In addition to individual training, a training curriculum will be developed. Continuous with the pilot curriculum set in Fall 2018 ROB
Optimization of the program efficiency
6. To optimize the efficiency of implementing the QEDP, CED should:
  • 6a. Continue to integrate various departmental guidance tools;
CED will continue to integrate the various departmental orientations. While helping advisers in selecting projects based on current departmental and government priorities, such integration will lead to greater common understanding and efficiency gain.

Two tools were recently revised and developed:
(1) "Integrator Tool 3.0", a revised tool in April 2018 to integrate the new departmental results and indicators identified in the Policy on Results;
(2) A dynamic new tool developed to explain how CED's 2018-2019 departmental priorities fit into the economic context of Quebec, the CED Strategic Plan, the Economic Growth Strategy for Quebec as well as the major government priorities.

CED will simplify and synthesize its future departmental guidance tools to address the concerns raised. In addition, during the biannual meetings organized with its Business Offices, CED will seize the opportunity to provide an update on the latest changes made to the departmental orientations and to survey its advisors on their appreciation of the tools at their disposal and their needs.
Spring 2019 PEIRB
6b. Document the cost effectiveness of using various types of contributions based on risk level and the assistance amounts allocated; In collaboration with the DECIA and the DGFM, CED will carry out a cost analysis by distinguishing between a refundable and non-refundable type of contribution. We will analyze repayable vs. non-repayable contributions since there are few funded projects of type A and D.

We will break down the costs according to the main stages of the life cycle of a contribution project.

The objective is that a cost is allocated to each phase of treatment. We will also discuss the cost of a simplified analysis vs a regular analysis and a low risk vs. higher risk analysis.
Winter 2019 Departmental Finances Branch (DFB)
6c. Document the efficiencies achieved further to establishing new business processes Over the last three years, CED has implemented several initiatives to optimize the internal approval process, for example: i) by implementing an automated risk profile integrated into its systems, ii) by using regular or simplified analyzes depending on the risk level and by implementation of CED’s risk management policy that includes risk management guidelines and parameters.

In addition, as part of the review of the program delivery processes to roll out the new G ° C management system, CED will review its business processes, including the aspect of e-signatures to document the possible gains and drawbacks.
Fall 2018 CEBID

Endnotes

Date modified: