Summative evaluation of the Municipal Rural Infrastructure Fund (MRIF) Program
Download the PDF version
About this publication
Publication author : Quebec Department of Municipal Affairs, Regions and Land Occupancy (MAMROT)
Collaborator : Advanis Jolicoeur, Canada Economic Development for Quebec Regions
Publish date : August 8, 2014
This report presents data collected as part of the Summative Evaluation of the Municipal Rural Infrastructure Fund (MRIF).
Table of Contents
This document presents data collected as part of the Summative Evaluation of the Municipal Rural Infrastructure Fund (MRIF). This study was carried out by the Advanis Jolicoeur research office on behalf of the Quebec Department of Municipal Affairs, Regions and Land Occupancy (MAMROT) and the Economic Development Agency of Canada for the Regions of Quebec (CED).
The data are presented in seven sections:
- Evaluation methodology
- Report on program achievements
- Program relevance
- Program implementation
- Program impact
Mandate delivery team
Nicolas Toutant, MPA, Evaluation: Project Leader
Kaddour Méhiriz, Ph.D., Evaluation: Consultant
Thomas Foulquier, Ph.D., Administration: Analyst
Carole Vincent, Analyst: Language Editor
Evaluation Sub-committee of the Management Committee
Christiane Jacques, Project Coordinator, MAMROT
Catherine Verge-Ostiguy, M. ATDR, Executive Assistant, MAMROT
Véronique Tremblay, Program Evaluation Coordinator, MAMROT
Claudia Dupont, Research Officer, MAMROT
Julie Chevalier, Analyst, Evaluation, CED
Éric-Pascal Ciaburro, Advisor, CED
In order to determine the relevance of the MRIF and the extent to which its objectives have been met, the Federal-Provincial Agreement provided for a summative evaluation of the program.
This evaluation establishes that:
- MRIF actions are consistent with government priorities;
- The program is relevant given changing infrastructure needs;
- Specifically, the program had a positive impact on investment and on the condition of the drinking water system;
- The roles and responsibilities of the levels of government involved in managing the MRIF are harmonized;
- Improvements are being considered with regard to administrative support for smaller municipalities, communication between program management partners, and performance monitoring.
The context of this evaluative study is set out in relation to the evaluation questions answered though this mandate.
1.1 Study context
The purpose of the Municipal Rural Infrastructure Fund (MRIF) was to “provide assistance for the rehabilitation, renovation, expansion, replacement or construction of water and wastewater infrastructure and for infrastructure work in support of local and regional development.”Footnote1.
The program results from the Canada-Quebec Agreement signed in July 2005 between the federal and provincial governments and runs until March 31, 2014.
The federal portion of the program is administered by the Economic Development Agency of Canada for the Regions of Quebec (CED) through a Memorandum of Understanding with Infrastructure Canada. Infrastructure Canada is accountable to Parliament for the MRIF.
The provincial share of the program is managed by the Quebec Department of Municipal Affairs, Regions and Land Occupancy (MAMROT). MAMROT’s mission is to “support the administration and development of municipalities, regions and the City of Montreal, promoting a sustainable and consolidated approach which benefits citizens.”Footnote2.
The Agreement is managed and administered by a management committee composed of two federal representatives and two provincial representatives, from CED and MAMROT, respectively.
The purpose of the MRIF is to improve Quebec’s community, urban and rural infrastructure and the quality of life of its residents, through investments intended to:
- Improve the quality of the environment;
- Support long-term economic growth;
- Improve community infrastructure;
- Build 21st century infrastructure using the best technologies and practices, as well as new approaches.
The MRIF includes three separate components, broken down as follows:
|Total Financial Assistance ($M)||Contribution From Each Government ($M)|
Source: Administrative data, MAMROT, 2013.
The first component, MRIF 1 – Drinking water and wastewater infrastructure, includes projects “making it possible to construct, renovate, improve or bring up to standard a municipal water pipe infrastructure, to solve a problem of drinking water quality or quantity or, in the case of sewers, to reduce the possible effects of effluents on drinking water sources and aquatic ecosystems, including fisheries resources and biodiversity, and to increase the effectiveness of the collection and treatment of waste waters, including rainwater, for the purpose of improving water quality, and for sustainable use and management of infrastructure and of water.”Footnote3.
Only municipalities are eligible under this first component, for which the financial assistance envelope set out in the Agreement is $143.1 million from each level of government.
The second component, MRIF 2 – Development of knowledge relative to municipal infrastructure, includes projects making it possible to design and implement approaches and tools to support the development of knowledge relative to municipal infrastructure and thus promote the cost-effective continuity of reliable infrastructure tailored to existing needs, for the purpose of protecting water.
Only municipalities are eligible under this second component, for which the financial assistance envelope set out in the Agreement is $2.0 million from each level of government.
The third component, MRIF 3 – Infrastructure supporting local or regional development, includes work to construct, renovate or improve service infrastructure supporting the maintenance or development of commercial, industrial or tourist activities; heritage or artistic infrastructure designed to preserve, develop and promote culture and heritage; or community sports recreational infrastructure designed to encourage a larger percentage of Quebecers to include sports and physical activity in their daily lives.
Municipalities and non-profit organizations (hereafter referred to as applicants) are eligible under this third component, for which the financial assistance envelope set out in the Agreement is $86.8 million from each level of government.
1.2 Evaluation mandate
The Federal-Provincial Agreement related to the infrastructure program provides for a summative evaluation of the MRIF in order to determine its relevance and the extent to which its objectives have been met. This evaluation will be sent to Infrastructure Canada once it is finalized.
Therefore, this evaluation involves a review of all program data from the early stages of its implementation to March 31, 2012. In addition, a formative evaluation regarding its implementation was completed in April 2008. This contract was subject to a public tender and was awarded to Advanis Jolicoeur on April 10, 2012.
1.3 Evaluation questions
As part of the MRIF evaluation, we are providing the answers to 12 evaluation questions from the tender specifications, addressing the program’s relevance, effectiveness and performance for each of the three components. We have outlined these questions in this section and have set out the collection methods used to answer them in Appendix I. This evaluation is based on four sources of data: a telephone survey of the perceptions of municipality and non-profit organization representatives; qualitative interviews with managers and officers involved in the program’s implementation; a documentary analysis; and an administrative data analysis using econometric models.
Question 1: What are the current local infrastructure needs in urban and rural regions? (methods: survey, qualitative interviews and documentary analysis)
Question 2: To what extent is the MRIF still relevant given the changing infrastructure needs and other government actions taken? (methods: survey, qualitative interviews, documentary analysis and administrative data analysis)
Question 3: Are MRIF actions consistent with government priorities? (methods: qualitative interviews and documentary analysis)
Question 4: Are the roles and responsibilities of the levels of government harmonized with MRIF actions? (method: qualitative interviews)
Question 5: Were MRIF communications appropriate for reaching the target groups? (methods: surveys and qualitative interviews)
Question 6: To what extent did the MRIF produce the desired immediate, intermediary and final results? (methods: administrative data analysis, survey and qualitative interviews)
Question 7: Did the MRIF have any unintended impacts? (methods: qualitative interviews)
Question 8: What were the barriers to the program’s success? How were they overcome? (methods: survey and qualitative interviews)
Question 9: What facilitated completion of the program? (method: qualitative interviews)
Question 10: To what extent do the two departments’ information management systems support the MRIF’s management and performance measurement? (method: qualitative interviews)
Question 11: Is performance information appropriately monitored? (method: qualitative interviews)
Question 12: Can the same results be achieved at lower cost? (method: qualitative interviews)
2. Evaluation Methodology
As mentioned previously, the data collection and analysis methods used for this evaluation were documentary analysis, a telephone survey, qualitative interviews and econometric models for analyzing administrative data. These methods are described in the following sub-sections.
The methodology used for this evaluation was selected in order to leverage the analysis of the CED and MAMROT databases and documents. This evaluation was also based on previous infrastructure program evaluations, including the formative evaluation of the MRIF, and on data from Statistics Canada and the Institut de la statistique du Québec.
2.1 Documentary analysis
The documentary analysis was carried out using reference documents relevant to understanding the program. It provided a portrait of program developments, drawn from the program’s description in the MRIF Rules and Standards Guide (2nd edition, 2007), its formative evaluation (2008), the reports of the Société de financement des infrastructures locales du Québec (March 2012), and the Report of the Sustainable Development Commissioner of Quebec – Auditor General of Quebec’s Report to the National Assembly for 2012-2013.
The econometric analysis of the program results carried out using administrative data is described further on in the analysis model. Note that it was developed based on the documentary analysis, which outlines the inputs and certain outputs of the program.
Sources used to complete the documentary analysis include MAMROT and CED project files.
2.2 Telephone survey
The program was evaluated through a telephone survey with target MRIF clients, which allowed for the statistical processing of the responses. The survey was developed by the Advanis Jolicoeur team in cooperation with representatives of the Evaluation Sub-committee.
2.2.1 Data collection preparation
The population is all of the potential participants, or the organizations eligible for funding. The population includes 1,111 municipalities and an unknown number of NPOs for component 3.
The sample consists of the organizations that we attempted to contact as part of this study, that is, 677 projects by municipalities or NPOs (some submitted more than one project).
Finally, completed interviews, or questionnaires completed by the respondents, are interviews conducted with individuals in the sample who agreed to participate; there were 371 respondents.
2.2.2 Sample composition
The sample is divided into two groups (see table below):
- The 1st group was intended to reach MRIF respondents, that is, those whose projects were approved or refused.
- This sample excludes applicants whose projects were cancelled or abandoned.
- The 2nd group of the sample was intended to reach municipalities that did not apply to the MRIF, so as to gather their views on topics such as their needs, why they chose not to participate, etc.
- This sample excludes municipalities with populations of under 200 residents, as their needs are less likely to relate to the program.Footnote4.
In addition, as an applicant may submit more than one MRIF application, projects were selected from within the sample so as not to have to contact a respondent for more than one project per component.
- The most recent project under each component was selected for each respondent (there may be more than one respondent per municipality or NPO).
- Therefore, respondents may have had to answer questions regarding more than one component of the questionnaire if they submitted projects under a number of components.
|1st group: Applicants||2nd group: Non-applicants|
|Refusals and Applications Received||Component 1||Component 2||Component 3||No Application||Sample Total|
|Single potential respondents from the SAMPLE||403||115||23||66||70||677|
|COMPLETED INTERVIEW respondents||225||66||12||33||35||371|
Source: Advanis Jolicoeur, 2012.
The telephone questionnaire was developed in cooperation with the Evaluation Sub-committee to answer the various evaluation questions outlined previously. The questionnaire included a section for each component of the program, a section for respondents whose projects were refused, and a section for municipalities that did not apply to the program.
2.2.4 Telephone data collection
Quantitative data was collected from September 7 to October 3, 2012. A total of 371 interviews were conducted during this period. The overall response rate for the telephone data collection was 54.8%.
2.3 Qualitative interviews
A series of qualitative interviews was carried out with managers involved in the implementation of the program. The objective of these interviews was to ensure a thorough analysis of the issues that could not be covered quantitatively. The grid was designed for managers and officers involved in the program and developed in cooperation with the Evaluation Sub-committee.
Qualitative interviews were conducted with six people between September 21 and October 10, 2012. Respondents were selected from among the people (two managers and four officers) who participated in managing the program. Employee turnover was a challenge in carrying out the qualitative interviews.
The interviews lasted an average of 25 minutes. They were recorded and summarized for analysis purposes. The interviews were conducted by evaluation question, which allowed for a question-by-question analysis, incorporating each respondent’s nuances.
2.4 Econometric models
This methodology is based on three econometric models used to analyze the administrative data: an analysis model of the determinants of access to funding, an analysis model of the program’s impact on municipal infrastructure investment, and an analysis model of the MRIF’s impact on intermediary and final targets. See Appendix II for a description of the models and details on the variables used.
2.5 Evaluation limitations
The first limitation of this summative evaluation relates to the quality of some of the available data. A number of variables planned for the econometric analysis of the administrative data had to be withdrawn, as the available data were incomplete. Therefore, it cannot be said that the statistical model used took into account all of the factors likely to have influenced the results. However, even if a theoretical model cannot control all of the factors that may influence the results, it nevertheless provides for a valid analysis of the selected areas of focus.
In addition, the effects of component 3 are largely undocumented in the evaluation, since very few data sources could be used to measure those effects objectively. This is a second significant limitation of the evaluation, as investments were substantial.
A third limitation of the evaluation, relating to the telephone survey, was that many projects had been carried out a number of years prior to the survey. Therefore, a number of respondents no longer worked for the applicants, which reduced the opportunity to obtain information on a large number of projects and the quality of the information collected. More specifically, few respondents could be contacted (N=11) from all the NPOs whose applications were accepted, which made it impossible to carry out a separate results analysis.
Finally, the lack of data on project performance is a fourth limitation of the study. For example, the evaluation of project performance in relation to certain environmental impacts is limited. The performance indicators identified at the start of a program, such as the MRIF, would certainly address this limitation in projects.
3. Report on project achievements
This section outlines the report on project achievements, taking into account its implementation context.
3.1 MRIF implementation context
CED’s mandate is “to promote the long-term economic development of the regions of Quebec”Footnote5 and its authority allows it to “plan and implement mechanisms to facilitate cooperation and concerted action with Quebec and communities in Quebec.” As part of this program, CED acts as a federal delivery partner and is mandated by Infrastructure Canada to deliver this program in Quebec. MAMROT’s role in terms of municipal infrastructure is also related to its mission, which is to “support the administration and development of municipalities, regions and the City of Montreal, promoting a sustainable and consolidated approach which benefits citizens.Footnote6.
These mandates are carried out within the budget frameworks set by the governments. The 2003 federal budget detailed the strategic investments intended to promote the economic viability of Canadian communities and to help Canadian cities and communities. To assist communities, the budget set aside an additional $3 billion to improve infrastructure over the next ten years, including $1 billion for municipal infrastructure.
In addition, in his budget speech of April 21, 2005, Quebec’s Finance Minister noted that “modernizing infrastructure throughout Quebec is an imperative”Footnote7 and announced that, for 2005-2006, investments totalling $4.9 billion would be made in infrastructure, both to maintain existing assets and to build new infrastructure.
One of the fundamental reasons for the implementation of the infrastructure programs of recent years is that the need for investment in municipal infrastructure exceeds the financial capacity of a number of municipalities.
In addition, the Government of Quebec’s Water Policy (PNE),Footnote8 implemented in 2002, is also directly linked to the implementation of infrastructure programs. The purpose of the policy is to “guarantee the sustainability of this resource, to protect public health and aquatic ecosystems, and to manage water from a perspective of sustainable development.”
Finally, in 2007, the Government of Quebec adopted the Act to promote the maintenance and renewal of public infrastructures, which allows for the identification and assessment of municipal infrastructure investment needs. Under this Act, the government must review and adopt, on an annual basis, an infrastructure investment plan and set out the budgets allocated to maintaining assets and reducing the maintenance deficit.
Under this Act, MAMROT adopted, in 2008, its first management framework, in which:
- The estimated value of Quebec’s municipal water infrastructure is $75 billion, including
- $7 billion for drinking water treatment facilities
- $8 billion for wastewater treatment facilities
- $60 billion for drinking water and wastewater systems
- The assessment of the annual investment required to maintain municipal water infrastructure is based on accepted investment rules:
- for drinking water and wastewater facilities: 2.5% of their value
- for underground drinking water and wastewater systems: 1% of their value
- The deficit to be eliminated was assessed at:
- $1.4 billion for the treatment of drinking water to complete the upgrading of municipal drinking water facilities, solve the problem in the Côte-Nord municipalities, and remedy the status of private drinking water distribution systems
- $800 million for the treatment of wastewater to comply with the commitments of Quebec’s Water Policy and to purchase phosphorous removal equipment
- $6.8 billion for underground municipal drinking water and wastewater systems
In addition, the management framework does not consider the need for local and regional development support infrastructure other than water infrastructure.
MAMROT is responsible for analyzing, selecting, optimizing and approving MRIF applicants’ infrastructure projects. The MRIF is one of many financial assistance programs managed by MAMROT since 1978 intended to meet these infrastructure maintenance and deficit reduction needs (Table 3). This contextual element is important to remember in analyzing the results of this MRIF evaluation, as it is difficult to separate its impact (c.f. section 6) from that of MAMROT’s other financial assistance programs to establish the clear benefits of the MRIF.
Other joint Canada-Quebec infrastructure programs managed by MAMROT include:
- Building Canada Fund – Quebec (BCFQ)
- Gas Tax Fund and Quebec’s Contribution (TECQ)
- Canada-Quebec Infrastructure Works (CQIW 1994, 1997, 2000)
- Infrastructure Stimulus Fund (ISF)
- Recreational Infrastructure Canada (RInC)
- Pipeline Renewal Program (PRECO)
- Canada Strategic Infrastructure Fund (CSIF)
In addition to these programs, MAMROT manages Quebec-specific initiatives such as the Quebec-Municipalities Infrastructure Program (PIQM).
|Program||Work Category||Financial Assistance Envelope ($M)||Quebec’s Contribution ($M)||Canada’s Contribution ($M)||Implementation Year|
|Quebec Wastewater Treatment Program (PAEQ)||Water||6,207.6||6,207.6||0.0||1978-1979|
|Municipal Wastewater Treatment Program (PADEM)||Water||541.3||541.3||0.0||1994-1995|
Canada-Quebec Infrastructure Works 1994 (CQIW 94)
|Water and economic development||1,187.1||662.1||525.0||1994-1995|
|Financial assistance program for the rebuilding of municipal infrastructure (PARIM)||Equipment repair||131.8||0.0||131.8||1996-1997|
|Canada-Quebec Infrastructure Works 1997 (CQIW 97)||Water and economic development||131.5||65.7||65.7||1997-1998|
|Financial assistance program for the repair of equipment damage caused by the ice storm of January 1998 (PREMEV)||Equipment repair||60.0||60.0||0.0||1998-1999|
|Running Water Program (PEVQ)||Water||180.0||90.0||90.0||1998-1999|
|Canada-Quebec Infrastructure Works 2000 (CQIW 2000)||Water||407.9||204.0||204.0||2001-2002|
|Urban and Village Renewal Program (PRUEV)||Economic development||90.0||45.0||45.0||2002-2003|
|Municipal Rural Infrastructure Fund (MRIF)||Water||290.2||145.1||145.1||2005-2006|
|Building Canada Fund – Quebec, Major Projects Component (BCFQ MP)||Water||497.2||248.6||248.6||2007-2008|
|Building Canada Fund – Quebec, Communities and Large Urban Centres Component (BCFQ CLUC)||Communities (water, economic development)||420.0||210.0||210.0||2007-2008|
|Large Urban Centres (water, economic development)||400.0||200.0||200.0||2007-2008|
|Canada Strategic Infrastructure Fund (CSIF)||Water||190.0||95.0||95.0||2008-2009|
|Infrastructures Québec (IQ)||Water||175.0||175.0||0.0||No data|
|Building Canada Fund – Quebec, Communities Component (BCFQ TOP UP)||Economic development||232.0||116.0||116.0||2009-2010|
|Quebec Pipeline Renewal Program (PRECO)||Pipelines||700.0||350.0||350.0||2009-2010|
|Infrastructure Stimulus Fund (ISF)||Water||71.6||35.8||35.8||2009-2010|
|Gas Tax Program and Quebec’s contribution 2006-2009 (TECQ 2006-2009)||Water||1,315.6||387.5||928.1||2006-2013|
|Quebec-Municipalities Infrastructure Program (PIQM)||Water||1,874.7||1,874.7||0.0||2002-2013|
|PRIME (technical expertise)||25.0||25.0||0.0||2012-2013|
|Recreational Infrastructure Canada Program (RInC)||Economic development||76.2||38.1||38.1||2009-2010|
|Gas Tax Fund and Quebec’s Contribution 2010-2013 (TECQ 2010-2013)||Water, studies, local roads and other infrastructure||2,102.8||616.0||1,486.8||2010-2011|
Source: MAMROT internal data
It should be noted that, according to the Quebec Auditor General’s Report to the National Assembly on Financial Assistance for Municipal Water Infrastructure (2013), although over $8 billion has been invested in municipal wastewater treatment infrastructure since 1978, an evaluation of the real impact of these actions in relation to the resources invested remains to be done.
This section provides a report on the MRIF program for the period covered by the evaluation.
Table 4 shows that a total of 890 projects were submitted and 233 projects were accepted, for an acceptance rate of 26%. This low rate is due to the fact that funds were limited. Nevertheless, without knowing how many, we know that some applications were redirected to other sources of funding.
Most of the proponents of the 233 projects were municipalities, and 19 projects were carried out by NPOs. In addition, 82% of contributions were paid to municipalities with 250,000 or fewer residents, for $182.5 million per government, federal and provincial, out of a contribution amount of $221.4 million.
|MRIF Component||Total Financial Assistance Envelope ($M)||Contribution per Government ($M)||Applications Received||Applications Accepted (Promises of Assistance)|
|No.||Value of Work Submitted($M)||No.||Maximum Eligible Cost ($M)||Quebec’s Contribution ($M)||Canada’s Contribution ($M)||Contributions Total ($M)|
Source: MAMROT, Administrative data sheet, January 21, 2013.
As of January 2013, through the MRIF, the provincial and federal governments each provided $221.4 million in financial assistance, in support of the beneficiaries’ contribution of $202.6 million.
The overall gap between the per-government contribution set out in the Agreement and the promised assistance is explained in part by program management costs and some projects’ lower than expected completion costs.
3.2.1 Achievement examples
Below are a few examples of projects funded by the MRIF.
In December 2012, the City of St Jérôme completed its filtration plant upgrading work, which required an investment of $12 million. Under component 1 of the MRIF, the governments of Quebec and Canada each contributed $3 million, for total government assistance of $6M. A number of jobs were required to meet the requirements of the Regulation respecting the quality of drinking water (RQEP), including installing an ultraviolet water disinfection system and increasing the capacity of the drinking water reserve. These upgrades allowed the City of St Jérôme to provide its residents with better drinking water and improved quality of life.Footnote9.
Other examples of projects under component 1:
- Upgrading of drinking water wells on Rang Ste-Marguerite in Mirabel. The maximum eligible cost of the project was $72,595.
- Renewal of systems in downtown Joliette. The maximum eligible cost of the project was $5 million.
In February 2012, the Centre Eau, Terre et Environnement of the Institut national de la recherche scientifique (INRS) published a study on changes in the residual thickness of cast iron water mains. The study’s maximum eligible cost is $150,000.Footnote10.
Another example of a project under component 2:
- Development of a guide for managing underground water supply facilities in Trois Rivières. The maximum eligible cost is $150,000.
To promote sports and community activities and to improve both community infrastructure and the wellbeing of residents, the Société sportive et sociale de Saint-Sylvestre carried out work to build a multipurpose community centre in St Sylvestre at a cost of $1,608,885. This centre can be used as a reception hall and a venue for year-round sports activities. The Société sportive et sociale de Saint-Sylvestre received government financial assistance of $1,072,590 under component 3 of the MRIF.
Another example of a project under component 3:
- Restoration of a lookout for the MRC Des Appalaches tourism office. The maximum eligible cost of the project is $49,725.
3.2.2 Results in relation to program objectives
To establish the link between program results and objectives, it is necessary to recall the objectives as defined and described in the program guide (see Logic Model, Appendix III). The explicit objectives of the MRIF were to obtain the following results in the short term for the three components of the program:
- Component 1Footnote11
- Upgrade drinking water treatment facilities
- Find a solution for a lack of drinking water
- Solve environmental contamination problems
- Reduce the frequency of wastewater overflow incidents
- Meet requirements on existing wastewater treatment facilities
- Improve public safety
- Upgrade mains and obsolete infrastructure
- Component 2
- Determine infrastructure condition and needs
- Establish priorities for action
- Develop planning and information tools
- Component 3
- Add or consolidate businesses
- Add or consolidate services to the public or to businesses
- Create or maintain long-term direct jobs
- Create or consolidate tourist/cultural/heritage hub
- Provide public access to recreational and sports infrastructure
- Provide use of infrastructure for communities
Without being able to give an exact figure, the analysis of the textual project description data shows that projects under component 1 often involved upgrading drinking water treatment and distribution systems and treating wastewater. Under component 2, for the purpose of developing action plans, most projects involved detecting and diagnosing the condition of water mains. Component 3 of the MRIF funded a wider range of projects, such as projects to build or repair recreational or community infrastructure.
Figure 1 shows the project funding breakdown by MRIF component.
Figure 1: MRIF Project Funding by Component (Millions of Dollars)
3.2.3 Quebec water policy
Continued water clean-up and improved management of water services is the fourth orientation of the Quebec Water Policy (PNE), in relation to Quebec’s wastewater treatment program (PAEQ) of 1978.
Of the policy’s four action areas, the MRIF more specifically relates to the areas that involve municipal clean-up and ensure the sustainability of municipal infrastructure while improving water services management.
The PNE commitments relating to MRIF projects are:
- Commitment 17: To provide financial assistance for bringing all drinking water supply and treatment facilities up to standard.
- Commitment 38: To complete the wastewater treatment program in small municipalities so as to eliminate the direct discharge of wastewater into waterways.
- Commitment 43: To induce all municipalities to achieve an infrastructure renewal rate of 0.8% per annum by 2007 and a rate of 1% per annum by 2012.
- Commitment 48: To increase Québec’s expertise in water services by promoting the use of new technologies and best practices.
It should be noted here that component 2 of the MRIF is directly linked to commitment 48, as it is aimed exclusively at projects for developing expertise in municipal infrastructure. In addition, a given project could relate to more than one commitment, and projects under component 3 of the MRIF have no link to the PNE.
Therefore, of the 233 projects funded under the MRIF, 97 were not related to any PNE commitment. The other 136 projects are divided as follows:
- 50 projects are related primarily to commitment 17, for a total investment of $159.1M, or an average investment of $3.182M per project (three levels of government).
- 27 projects are related primarily to commitment 38, for a total investment of $111.7M, or an average investment of $4.137M per project.
- 22 projects are related primarily to commitment 43, for a total investment of $48.6M, or an average investment of $2.21M per project.
- 37 projects are related primarily to commitment 48, for a total investment of $5.2M$, or an average investment of $0.14M per project.
Figure 2 shows the breakdown of the 233 MRIF-funded projects, by PNE commitment.
Figure 2: Breakdown of MRIF-funded Projects by PNE Commitment
Therefore, the $324.6M in investments under components 1 and 2 of the MRIF relating to the PNE shows that water management is a shared concern.
4. Program relevance
This chapter deals with the relevance of the MRIF program, that is, its capacity to meet the priority needs of municipalities. Relevance will be assessed taking into account possible changes in infrastructure support needs since the program was designed. Remember that the MRIF is implemented and administered by CED, Infrastructure Canada and MAMROT, as it relates to the mandates and mission of these departments, as well as the needs of municipalities.
The MRIF was and is still linked to the federal and provincial government infrastructure priorities set out in various budgets. The 2003 federal budget announced a strategic investment of $3 billion over 10 years to promote the economic viability of Canadian communities and to assist cities and communities. In addition, in 2005, Quebec’s Finance Minister announced a $4.9 billion investment in infrastructure building and maintenance. Finally, both governments recently announced major investments in infrastructure, attesting to persisting needs in this area.Footnote12.
4.1 Infrastructure needs
The first part of this portrait is based on the applicant survey, completed by qualitative data from the interviews with department representatives. As shown in the table on the next page, the municipalities were asked to rate the extent of their needs for component 1 of the MRIF. Generally speaking, municipalities consider drinking water and wastewater main rehabilitation and repair needs to be the most important.
|Rehabilitating or replacing the drinking water distribution system||Rehabilitating or replacing the wastewater collection system||Expanding, implementing or upgrading the drinking water treatment system||Expanding, implementing or upgrading the wastewater treatment system|
|Do not know||0||0||0||0|
Source: Advanis Jolicoeur, Survey of municipalities (Sept. - Oct. 2012): qc11a -qc11b -qc11c -qc11d
In the following table, the contacted municipalities rated changes in the overall condition of their infrastructure in the last five years. In general, the municipalities said that the overall condition of their infrastructure improved.
The percentage of municipalities whose infrastructure deteriorated is relatively small. Wastewater treatment systems are an exception, however, as a larger number of municipalities said that the condition of their infrastructure remained stable or deteriorated.
|Drinking water distribution system||Wastewater collection system||Drinking water treatment system||Wastewater treatment system|
|(2) Remained stable||15%||25%||17%||29%|
|(4) Does not apply||12%||11%||26%||15%|
|(4) Do not know||0||1||0||0|
Source: Advanis Jolicoeur, Survey of municipalities (Sept. - Oct. 2012): qc12 a, b, c, d.
A smaller proportion of municipalities that did not apply to the MRIF also report improvement or stability in the condition of their infrastructure during this period (Table 14, Appendix IV). Despite this positive view of the overall condition of infrastructure, the available studies and representatives of the departments surveyed find that needs in this area are still high.
For component 2 of the MRIF, the issue of relevance is treated in a different way. The objective of this smaller component of the program is to improve practices and diagnose the condition of infrastructure. Representatives of the departments consider it to be relevant given the potential for results and efficiency that these projects may bring to the municipalities in the medium or long term. Enhanced knowledge of the condition of infrastructure is key to ensuring a long-term investment plan tailored to the needs of the municipalities.
Finally, for all components of the MRIF, many of the applicants who were asked how the support provided by the program aligned with their own needs said that the program met their needs adequately. 71% of respondents said that the MRIF met their needs very well and 28% said that the program met their needs rather well (Table 7). Therefore, respondents are very satisfied with the program’s coverage of the work.
|(1) Very well||71%|
|(2) Rather well||28%|
|(3) Rather poorly||1%|
|(4) Very poorly||0%|
|Do not know||1|
Source: Advanis Jolicoeur, (Components 1,2,3) Survey of municipalities (Sept. - Oct. 2012): Q17 – Overall, would you say that the program met your organization or your municipality’s needs very well, quite well, rather poorly or very poorly? 2012.
It should be noted that the results were essentially the same for municipalities that did not receive financial assistance.
4.2 Complementary studies on the evaluation of infrastructure needs
A number of studies have documented the lack of investment in municipal infrastructure, including:
- Deloitte & Touche (2012),Footnote13 carried out for the Union des municipalités du Québec, which assesses the infrastructure deficit in Quebec municipalities at $34.2 billion. This study takes all infrastructure into account, including road infrastructure not covered by the MRIF program.
- The MAMROT management framework (2008), in which the drinking water treatment deficit was assessed at $1.4 billion, the wastewater treatment deficit at $800 million, and the municipalities’ underground drinking water and wastewater systems deficit at $6.8 billion, for a total deficit of $9 billion.
- A recent study by the Canadian Construction Association,Footnote14 in which a Canada-wide drinking water and wastewater infrastructure needs assessment was carried out. Infrastructure needs remained substantial in Canada as a whole. Specifically, the replacement value of wastewater treatment infrastructure in “fair” to “very poor” condition was $39 billion, while that of drinking water treatment infrastructure in “fair” to “very poor” condition was estimated to be $25.9 billion.
- It should be noted that no studies were identified on investments under component 3 (infrastructure supporting local or regional development).
Interestingly, the Deloitte & ToucheFootnote15 study indicates that increased government program investment has significantly reduced infrastructure deficit growth. The infrastructure investment deficit grew by 5.7% over the 2003-2007 period, while it increased by just 1.8% from 2008 to 2011. Despite this reduced investment deficit growth, infrastructure will continue to deteriorate.
The findings of these studies are therefore the same: municipal infrastructure needs are high.
4.3 Other programs
Although all of the stakeholders feel that needs are still high and that these needs relate to the areas covered by the program, the MRIF is a relatively small program that could not meet all of these needs. However, it is part of a suite of federal and Quebec municipal infrastructure funding programs.
Therefore, with regard to the relevance of the MRIF program in relation to other infrastructure programs, most of the survey respondents said that, in general, other sources of funding were based on similar, if not identical, criteria. The main reason for these apparent duplications is that the programs are of a limited duration and that they overlap one another to cover the range of infrastructure needs. Therefore, there does not seem to be duplication, but rather complementarity of funding tools with similar characteristics, following and sometimes overlapping one another. Lastly, it should be noted that the objective of the MRIF was to target rural communities, which sets it apart from the other programs.
5. Program implementation
The implementation analysis focused on the following issues: government roles and responsibilities, MRIF communications, project delivery, access to funding and performance management and measurement.
5.1 Government roles and responsibilities
The Agreement management committee is responsible for
- the implementation of the Agreement;
- ensuring compliance with its terms and conditions;
- monitoring the program (deadlines, reports, specific decisions, financial compensation and the eligibility of particular projects); and
- day-to-day management, including the management of specific cases.
Based on the information obtained from the individuals interviewed, the roles and responsibilities appear to be well aligned and quite efficient, since, from an operational standpoint, the MRIF seems to provide continuity with previous government funding programs. It was noted, however, that communication could be improved to simplify monitoring. Relevant information was sometimes slow in reaching the various players. This issue primarily affected the representatives of the various departments involved in the accountability process.
MAMROT’s role is to oversee the implementation of the program and the selection of projects for approval. Quebec authorities are responsible for most of the program management tasks.
CED is responsible for monitoring the use of funds on behalf of the federal government. Its role is to ensure that projects comply with the standards in the Framework Agreement and the terms and conditions of the program.
5.2 MRIF communications
In terms of communications, it was noted that the majority (66%) of the municipalities that did not submit applications (n=35) knew about the MRIF program; however, a significant proportion (34%) of these municipalities were unaware of the program (n=35). Details regarding the methods used by the program proponents to reach the target clientele were provided in the formative program evaluation (p.36):
An official announcement of the MRIF was made on September 21, 2005, and a guide on program rules and standards was sent out to all municipalities in Quebec. Since then, public announcements have been made for projects registered under the Agreement by the governments of Quebec and Canada, in conjunction with the proponent municipality or organization. In addition, the MAMR, Agency and Infrastructure Canada websites post information about the programs. All of the announced projects are also posted on the MAMR and Agency websites.
Considering these sources of information about the program, 75% of the respondents who had submitted an application under the MRIF said they were well informed by MAMROT as to the existence of the program (Table 16, Appendix IV). The various sources cited were the MAMROT’s website and its bulletins and emails. It was noted, however, that departments had trouble reaching certain client groups, such as municipalities with limited administrative capacity.
Finally, it should be noted that a significant number of those who applied for funding, but whose projects did not receive financial assistance, were never informed that their project had not been selected under the program. According to MAMROT representatives, since applications that were ineligible for MRIF funding were transferred for analysis under other programs, applicants could not systematically deduce from response times that their initial application had been rejected.
5.3 Project Implementation
MAMROT supports applicants in carrying out their projects, with a view to optimizing their applications. Survey data show that, for 46% of respondents, the greatest support was provided during the funding application phase, whereas for another 25% of the respondents (n=91), it was during expenditure monitoring (Table 20, Appendix IV).
All in all, 87% of applicants were generally satisfied with MAMROT’s support. On the basis of these responses, MAMROT appears to have provided high quality support for applicants who obtained funding to carry out their projects (Table 8).
|(1) Very satisfied||40%|
|(2) Rather satisfied||47%|
|(3) Rather dissatisfied||11%|
|(4) Very dissatisfied||2%|
Source: Advanis Jolicoeur, Survey of municipalities – components 1, 2 and 3 (Sept.–Oct. 2012): Q16b – Would you say you were very satisfied, rather satisfied, rather dissatisfied or very dissatisfied with the support obtained from MAMROT?
When questioned about their general impressions regarding the monitoring required by MAMROT, 78% of respondents stated that the monitoring of their project had been relatively easy (Table 18, Appendix IV).
During the qualitative interviews, a number of managers from the departments in question raised the issue of cost and schedule overruns. It should be noted that this same issue was addressed in the Report of the Auditor General of Québec to the National Assembly (2013), in the section on Financial Assistance for Municipal Infrastructure, as follows: [Translation] “Although such discrepancies may sometimes be the result of particular situations (for example, other major projects launched at the same time, which leads to an increase in labour and material costs), they may also be an indication of a market failure that undermines free competition.”Footnote16
According to the findings of the survey of applicants conducted as part of this evaluation, 27% of the projects resulted in cost overruns, which were assumed in 86% of the cases by the municipality. Tables 9 and 10 detail the reasons for the cost overruns, and their impact on the projects. However, even though certain projects may have reported cost overruns, it is important to note that this had no impact on the overall program envelope. In fact, the total amount of funds spent was slightly less than the allocated envelope.
Specifically, Table 9 shows that, in the case of over half of the applicants, the cost overruns were caused by an increase in the quantity of work in relation to what was initially planned and a rise in resource prices.
|a) Increase in the quantity of work||b) Technical constraints requiring the use of more costly processes or material||c) Rise in resource prices (material, energy, labour, etc)||d) Unforeseen events (cold weather, snow, etc.)||e) Higher bids than expected|
Source: Advanis Jolicoeur, 2012. Did the following factors contribute to the cost overruns?
|a) Reduction in the volume of work||b) Increase in the municipality’s share||c) Increase in the MRIF contribution||d) Other sources of funding were obtained|
Source: Advanis Jolicoeur, 2012. Did the cost overruns have the following impacts on your municipality?
Finally, with respect to problems relating to the carrying out of the projects, 26% of applicants indicated that they had trouble obtaining environmental approval, while 21% experienced difficulties during the onsite audits conducted by MAMROT (Table 19, Appendix IV). It should be noted that, given the binary nature of the questions (yes or no answers), it is impossible to qualify these problems.
5.4 Access to funding
MAMROT project managers assess funding applications using a compliance assessment sheet. On the basis of program criteria, they then award a priority rating that corresponds to the level of the issue presented, and they check to make sure the application is properly documented. Projects with a priority level of 1 to 4 are then subject to a detailed analysis, after which a funding recommendation is submitted to the authorities. Selected projects will receive a promise of financial assistance.
The analysis of administrative data reveals that 46% of municipalities did not submit funding applications under the MRIF. Of those that did submit projects, 31% obtained funding (it should be remembered that the same municipality may have submitted more than one project).
In this evaluation, we sought to understand the reasons behind access to funding under the MRIF. The data analysis, which was based on an econometric modelFootnote17 that took into account a number of variables, such as the size of the population and the region, revealed that the smallest municipalities were less likely to submit applications for funding under the MRIF. With respect to the awarding of financial assistance, the econometric analysisFootnote18 also showed that the smallest municipalities, as well as those with limited administrative capacity,Footnote19 were less likely to obtain funding under this program. These issues were also raised in the CQIW 2000 evaluation (pp. 82–83).
Since each level of government generally assumes one third of the project costs, municipalities that are in a precarious financial position may, for this reason, be less inclined to submit applications.
However, it should be noted that, according to departmental representatives, it is possible for small municipalities to reduce the initial cost of participating in the program by entrusting the file preparation process to a private engineering consulting firm.
Those municipalities that did not submit applications were also questioned during the interviews. Survey data show that, in 35% of the cases, the main reason for not submitting an application was that the work that the municipality needed done was not eligible under the program (Table 17, Appendix IV).
In 25% of cases, the municipalities took advantage of another program to carry out their work. A further 15% of the municipalities indicated that they lacked the time or knowledge required to submit an application.
The issue of the capacity of small municipalities to submit projects was brought up with MAMROT representatives, who stated that they provided support for municipalities during the funding application process.
The funding access analysis seems to indicate that small communities and those with limited administrative capacity are less likely to submit quality applications, given their limited resources and despite the conditional preparation support provided by MAMROT.Footnote20. It would be interesting, within the context of a renewed municipal infrastructure program, to delve further into the reasons behind the small number of applications from this group of clients.
5.5 Performance management and measurement
A number of challenges to the optimal management of the program came to light during the qualitative interviews with MAMROT and CED representatives.
To start with, according to the respondents, the computer systems used (INFRA system) were not always adapted to project management. It should be noted that this issue had already been addressed in the final version of the MRIF formative evaluation.
Furthermore, the reporting requirements under the Agreement pose an administrative burden with regard to communications between the partners. Sometimes, within the same level of government, the structure of the Agreement makes it necessary to contact recipients to obtain information. There is a perception that it takes a long time to access information, and the partners’ data exchange formats are not always compatible.
Finally, a number of respondents mentioned the lack of performance indicators, an issue that had already been identified during the formative evaluation of the MRIF in April 2008. Not having a performance measurement system hinders project evaluation and, consequently, the evaluation of the program as a whole.
5.6 Implementation summary
The findings regarding the implementation of the MRIF are as follows:
- The roles and responsibilities of the various MRIF partners are well aligned. However, there is room for improvement relating to communication for reporting purposes among the representatives of the departments involved.
- Although the communication of the program, which was intended to target rural municipalities, made it possible to reach the applicants, the smaller municipalities were, statistically speaking, less likely to submit funding applications, and had a harder time obtaining funding.
- MAMROT’s support for municipalities during the entire MRIF funding application, analysis and monitoring process was widely viewed as satisfactory.
- The lack of performance indicators in the MAMROT program management systems was once again noted, as it had been in the MRIF formative evaluation conducted in 2008.
6. Program impact
This section deals with the impact of the program, ie, the results, the unintended repercussions and the potential for greater efficiency. It should be remembered that the objectives of the MRIF are to:
- enhance the quality of the environment;
- support long-term economic growth;
- improve community infrastructure; and
- build 21st century infrastructure through best technologies, new approaches and best practices.
6.1 Immediate, short-term and long-term results
The focus here is on the impact of the MRIF on investment in, and the condition of infrastructure.
6.1.1 Impact on investment
The measurement of the impact of the MRIF on investment is based on two analytical approaches. The first part of this analysis was conducted using the applicant survey results; the second part relied on the secondary database.
In order to determine whether the program stimulated investment in municipal infrastructure, applicants were asked whether they would have carried out their MRIF-funded project if the program had not existed. A total of 19% of respondents stated that the project would have gone ahead all the same (n=111).
However, among those who stated that they would have carried out their project even if the MRIF had not existed, only 40% believed that their project would have taken place within the same timeframe (n=20).
In terms of the volume of work, of the 19% of respondents who indicated that they would have gone ahead with the work without the assistance of the program, 62% stated that the work would have taken place on the same scale (n=21). This suggests that approximately 12% (62% of the 19%) of the clients that submitted an application for funding under the MRIF would have gone ahead with their project, even without the program. However, 81% of respondents stated that the projects would not have taken place without MRIF support (n=111).
The second analytical approach for measuring the impact of the MIRF program is based on the investment data compiled by MAMROT. A database was created for analysis purposes using data from MIRF-funded projects, the municipalities financial reports and Statistics Canada census data. This database covers the period from 2005 to 2011.
The data analysis techniques (see the section on methodology and Appendix II) were used to estimate the impact of the government funding under the MRIF on municipal spending. In the case of a municipality that received funding in 2005, for example, we estimated the impact of this funding on the municipality’s investment spending in 2005 and 2006.
The analysis showed that the program has a positive and statistically significant impact on investment in municipal infrastructure. The increase in investment spending, both during the year in which municipalities obtained funding under the program and in the following year, was 20% greater in these municipalities than in other municipalities that did not receive funding.
However, the impact of the program on long-term economic growth is still unknown and was not measured in this evaluation.
6.1.2 Impact on the condition of component 1 project infrastructure
Half of all investments under component 1 involved work on wastewater collection, interception and treatment infrastructure, including sanitary and combined sewers. A quarter of the investments went toward drinking water treatment infrastructure, followed by drinking water distribution, drinking water supply and storm sewer infrastructure.
Three categories of indicators were used to measure the impact of MRIF component 1 on drinking water and wastewater systems: (1) the number of water main breaks; (2) the number of kilometres of drinking water and sewer pipes; and (3) drinking water and wastewater discharge standards.
The model usedFootnote21 to measure the impact of participation in MRIF component 1 on these indicatorsFootnote22 reveals the following:
- The MRIF led to a reduction in the number of water main breaks and an increase in the length of the drinking water system.
- The measured effect between the intervention and the length of the sewer system is not statistically significant. However, when interpreting this result, one should keep in mind that the amount of work that can be carried out under the MRIF represents a small investment compared with the volume of fixed assets that Quebec’s sewer system infrastructure represents.
- There is no statistically significant effect on compliance with drinking water standards, nor with wastewater effluent standards (which are measured in terms of levels exceeding bacteriological standards, trihalomethane levels in the water, and the ratings obtained at wastewater treatment stations and from overflow structures).
We should point out that, although it cannot control all the factors that could influence the results, a theoretical model nevertheless allows for a valid analysis of the selected areas of focus.
6.1.3 Impact of projects under components 2 and 3
All of the work under component 2 involved conducting inventories and diagnosing the facilities with a view to developing action plans. It is difficult to quantify the impact of the interventions in terms of improvements to practices or technologies. The development of best practices is a long-term process. With respect to diagnoses of the condition of the infrastructure, the analysis of the improvement of municipal investment plans remains a subjective exercise.
According to the representatives of the various departments, it is much harder to assess the impact of these projects, given the nature of the interventions, than it is to assess the impact of projects under the other components.
A broad variety of work was carried out under component 3, with an emphasis on sports facilities. This component allowed some municipalities to introduce enabling infrastructure, such as community centres or tourism facilities.
The impact of component 3 can only be measured from a qualitative standpoint. Department representatives noted that, in their view, component 3 had a positive impact on residents’ quality of life. Given the nature of the investments, the medium- or long-term use of this infrastructure cannot be determined at the current time. However, according to department representatives, investments under component 3, offer a potential for improvement to the residents’ quality of life. The quantitative results in the following section put the program results for component 3 recipients into perspective.
6.1.4 Short-term project results
The quantitative survey findings show the short-term results of the applicants’ projects under each MRIF component.
It was noted that, under component 1, the municipalities faced a significant number of environmental contamination issues. Half of the recipient municipalities were able to address these issues through the MRIF. Over half of the recipient municipalities were also able to upgrade pipes, obsolete infrastructure and drinking water treatment facilities (Table 11).
|A) Upgrade drinking water treatment facilities||B) Find a solution for a lack of drinking water||C) Solve environmental contamination problems||D) Reduce the frequency of wastewater overflow incidences||E) Meet requirements on existing wastewater treatment facilities||F) Improve public safety||G) Upgrade pipes and obsolete infrastructure|
Source: Advanis Jolicoeur, Survey of municipalities (Sept.–Oct. 2012): qc14, 2012.:
The majority of the municipalities that received funding under component 2 focused on developing planning and information tools for their infrastructure, rather than simply determining the condition of their infrastructure and their infrastructure needs (Table 12).
|Determine infrastructure condition and needs||Establish priorities for action||Develop planning and information tools|
Source: Advanis Jolicoeur, Survey of municipalities (Sept.–Oct. 2012): qd12, 2012.
Under component 3, the vast majority of the recipients added or consolidated services for the public and provided public access to recreational and sports infrastructure through their projects. Projects targeting tourism or the implementation or retention of commercial or industrial business undertakings were a minority (Table 13).
|A. Provide public access to recreational and sports infrastructure||B. Add or consolidate services for the public||C. Provide public access to cultural or heritage infrastructure||D. Create or maintain longterm jobs||E. Create or consolidate a tourist hub||F. Foster the implementation or retention of commercial or industrial business undertakings|
Source: Advanis Jolicoeur, Survey of municipalities (Sept.–Oct. 2012). qe14: Indicate whether or not the following apply to your 2012 project.
According to the municipalities, the projects under all three components resulted in the achievement of the targeted program results.
6.2 Unintended impacts
The surveys of applicants and the interviews with MAMROT representatives revealed some unintended impacts of the MRIF program.
Given the limited amount of funding available, and the fact that there was a deadline for submitting projects under the program, many municipalities had to submit their funding applications quickly, which meant that some applications were not perfectly prepared. This partially explains why the initial evaluation of the project delivery costs was not always accurate.
Overall, 27% of recipients experienced cost increases during the course of their project (n=107). It should be remembered, however, that in 86% of the cases the increase was funded by the municipality.
6.3 Potential for enhanced efficiency
In terms of improving the program’s efficiency, the respondents noted that it would help if the demand for construction services could be spread out over a longer period of time. If governments were more flexible with respect to deadlines, recipients would be able to carry out the work over a longer timeframe, which would reduce the price increases seen when the demand for construction services significantly exceeds supply. This cost increase issue is not directly related to the MRIF, given the modest scale of the program, but rather to the fact that a number of programs were in place at the same time.
The findings of the summative evaluation of the MRIF are as follows:
- The roles and responsibilities of the levels of government directly involved in the management of the MRIF—CED and MAMROT—are harmonized. However, communication between partners in terms of reporting could still be improved.
- MRIF actions are consistent with government priorities and the program is relevant given changing infrastructure needs and other government infrastructure investment actions.
- MAMROT’s support for municipalities throughout the MRIF funding application, analysis and monitoring process was widely viewed as satisfactory, but performance monitoring would benefit from the support of relevant indicators.
- Communications intended to inform target groups (municipalities and NPOs) about the MRIF were adequate.
- The smallest municipalities seem to have been less inclined to apply for financial assistance and appear to have had more difficulties in applying to this program. In addition, a number of small municipalities said that they did not have the resources or did not know how to apply. We suggest that the reasons for the lower rate of participation among these municipalities be explored further.
- The MRIF program stimulates investment and had a positive impact on the condition of the drinking water system. In contrast, in light of the available indicators, the program had no measurable impact on the condition of the wastewater system or on compliance with regulations on drinking water and wastewater management. Lastly, it is difficult to measure the impact of the program on the quality of life and economic development of communities.
- Footnote 1
- Footnote 2
- Footnote 3
- Footnote 4
Among the exclusions are 33 municipalities with under 200 residents, 14 northern villages and 96 unorganized territories. Source : Décret de population
- Footnote 5
- Footnote 6
- Footnote 7
- Footnote 8
- Footnote 9
- Footnote 10
- Footnote 11
Formative evaluation of the MRIF
- Footnote 12
- Footnote 13
Joint, Deloitte & Touche, Faits saillants de l’étude sur l’infrastructure municipale du Québec, September 2012, page 8.
- Footnote 14
Joint, Canadian Construction Association, Canadian Public Works Association, Canadian Society of Civil Engineering, and the Federation of Canadian Municipalities, Canadian Infrastructure Report Card, Volume 1, page 20, 2012.
- Footnote 15
Op. cit., Deloitte & Touche, page 9.
- Footnote 16
MAMROT’s response in the same report: [Translation] “The 41% increase in financial assistance [for municipal infrastructure projects funded by MAMROT between 2006 and 2012] is primarily attributable to changes in the project scope further to its full definition, the technical analysis and adjustments to meet MDDEFP regulatory requirements, as well as in order to better meet municipalities’ water needs, and because of inflation, given the relatively long timeframes for carrying out such projects. Part of the increase is also the result, in certain cases, of the granting of a higher assistance rate than that used at the time of the initial promise of funding, in accordance with regulations and standards.”
- Footnote 17
The results of the econometric analysis are available from the CED Planning and Evaluation Directorate, upon request.
- Footnote 18
- Footnote 19
Administrative capacity is defined based on the municipality’s “number of human resources” indicator.
- Footnote 20
MAMROT will reimburse consultation costs associated with the preparation of an application, provided the project is accepted and that the costs in question were incurred after the date on which the Canada-Quebec Agreement on the MRIF was signed. Applicable costs are limited to a maximum of 20% of direct eligible costs.
- Footnote 21
A quasi-experimental model using before (2005) and after (2011) measurements, and statistical controls.
- Footnote 22
The results of the econometric analysis are available from the CED Planning and Evaluation Directorate, upon request.
Appendix I: Survey Questions
|Telephone Survey of Applicants||Qualitative Interviews With Program Managers||Documentary Analysis||Analysis of Administrative Data|
Question 1: What are the current local infrastructure needs in urban and rural regions?
The objective of the analysis of the relevance of the MRIF was to determine whether the work and the eligible infrastructure was in line with the priority needs of the program’s target clientele. When conducting this analysis, we took into account the views of the program recipients, as well as those of MAMROT and federal stakeholders.
Question 2 : To what extent is the MRIF still relevant given the changing infrastructure needs and other government actions taken?
The analysis of the program’s relevance also took into account changing needs, as well as other infrastructure support programs. Available data on the condition of infrastructure, as well as on federal and Quebec government transfer programs, provided an idea of the changes in the problems and allowed for linking of the program’s various areas of focus. Recipients were asked whether the program met their needs, and about the program’s accessibility. In addition, when applicable, the MRIF was compared with other infrastructure programs to determine the complementarity or overlapping of government action.
Question 3: Are MRIF actions consistent with government priorities?
When the MRIF was designed, certain basic program objectives were adopted. These objectives were analyzed in light of current government priorities, by means of the documentary analysis. A second important aspect was also analyzed: even if the program was deemed to be in line with government priorities, it was necessary to ensure that achievements were consistent with these objectives. To this end, the work was analyzed based on the information in the administrative databases.
Question 4: Are the roles and responsibilities of the levels of government harmonized with MRIF actions?
To answer this question, interviews were conducted with stakeholders in the various departments involved. We also described players’ roles and responsibilities. This description allowed us to identify potential problems with respect to harmonization between the various levels of government, which could limit the program’s success.
Question 5 : Were MRIF communications appropriate for reaching the target groups?
To answer this question, interviews were conducted with applicants who submitted projects, regardless of whether the projects were accepted or rejected, as well as with a small sample of municipalities that did not submit applications. The means of communication by which they obtained program information were described. MRIF municipalities (components 1, 2 and 3) were also questioned about the reasons for not submitting an application. This information was used to determine whether there was a problem relating to program accessibility—above and beyond simply knowing about the program—which could have an impact on the program’s success.
Question 6 : To what extent did the MRIF produce the desired immediate, intermediary and final results?
Since the first measurable results relate to the impact of a program such as this on investment levels, the immediate results were analyzed in terms of changes in the level of investment. Intermediary results include the projects’ impact on the municipalities’ situation with respect to the condition of its infrastructure, the quality of its drinking water and compliance with wastewater discharge standards. Applicants were asked about changes in the condition of their infrastructure, and these changes were measured. Finally, applicants were questioned about the incentive nature of the program.
Question 7 : Did the MRIF have any unintended impacts?
In this type of program, the two main risks were the replacement of municipal expenditures with the MRIF (no additional impact) and the risk of developing overcapacity. These potential issues were analyzed based on available data and interviews with departmental experts and applicants. Open questions were also developed to determine whether the program had any other unintended impacts, including positive impacts.
Question 8: What were the barriers to the program’s success? How were they overcome?
To determine the barriers to the program’s success, various aspects were measured. First of all, the management of the program’s implementation was evaluated with the stakeholders and program managers. Applicants were also questioned about problems they encountered in order to find out how they overcame these barriers.
- Access to funding: Data pertaining to the municipalities’ socio-economic and financial situations, as well as to their administrative capacity, allowed us to conduct an analysis that compared municipalities that took part in the program with other eligible municipalities. We used the same logistic regression analysis model that had been applied in other evaluations.
- Capacity of the program to identify infrastructure requirements.
- Project implementation (carrying out the work provided for in the Memorandum of Understanding within the established timeline and without cost overruns).
Question 9 : What facilitated the delivery of the program?
This question specifically relates to the examination of the factors that facilitated the implementation of the program. We put this question to the various stakeholders and the program managers for the purpose of identifying the factors that resulted in the successful implementation of the program. Applicants were also questioned on this matter during the semi-directed interviews (quantitative component) and in the survey (qualitative component).
Question 10 : To what extent do the two departments’ information management systems support the MRIF’s management and performance measurement?
Quality data are an essential component of adequate management monitoring. The quality and content of the data were analyzed, and these analyses were examined from the perspective of management monitoring expectations prior to implementation in order to determine whether the results were in line with expectations. Furthermore, the analysis of the relevance of the management indicators was conducted in order to obtain a diagnosis of the situation.
Question 11 : Is performance information appropriately monitored??
The management processes were analyzed based on the information obtained from the interviews with the managers. Various aspects were described, including the capacity to analyze information, and monitoring methods. This analysis was conducted by looking at performance monitoring in terms of the quality of the available indicators. We also reviewed administrative data, including evaluation reports and other reports relating to MRIF performance monitoring.
Question 12 : Can the same results be achieved at lower cost?
To answer this question, we asked the recipients about the tendering processes associated with carrying out projects, difficulties obtaining contractors, and incentives for correctly targeting needs and ensuring sound management. We also looked at the resource optimization processes and control mechanisms in place. The views of the managers of this program at MAMROT and CED were also relevant in this regard.
Appendix II: Analysis model operationalization variables
Analysis model for access to funding determinants
The goal of this analysis was to determine whether the characteristics of certain municipalities meant that these municipalities experienced particular problems accessing MRIF program funding.
We used logistic regression techniques to analyze the factors that increased or decreased the likelihood that a municipality would submit an application for, and obtain, funding under the MRIF.
To carry out this analysis, we referred to the list of municipalities that had submitted projects under the MRIF and the municipalities that had obtained funding, as well as data relating to the municipalities’ socio-economic characteristics.
Analysis model of the impact of the program on investment in municipal infrastructure (direct targets)
For the purposes of this evaluation, panel data analysis techniques were used to evaluate the impact of the program on investment in municipal infrastructure. We estimated the impact of participation in the program (regardless of whether or not the municipality received funding) on the amount of investments in municipal infrastructure.
To carry out this analysis, we needed to know the amounts of the funding granted under the MRIF, as well as the amounts invested by municipalities in eligible infrastructure for the period from 2005 to 2010. We also used, as control variables, the municipalities’ socio-economic characteristics and any amounts obtained under other infrastructure funding programs during the same period.
Analysis model of the impact of the MRIF on intermediate and ultimate targets
As was noted in the mandate, we used the quasi-experimental specification with measurements taken before and after the program to analyze the impact of the program on the condition of drinking water and wastewater infrastructure and on the quality of drinking water and wastewater management. For each indicator, we compared the situation in 2005 with that in 2011, the goal being to measure the impact of municipalities’ participation on these indicators, while controlling external factors that could skew the analysis. As with the other analyses, we needed data for these indicators for the period from 2005 to 2011.
For the purposes of these analyses, we used data from four categories of variables. The data sources used include data on the MRIF (applications and funding offers made to recipients); the municipalities’ financial reports and management indicators; MDDEFP drinking water quality indicators; and Statistics Canada Census data.
Analysis model operationalization variables
Intervention variables are those that measure the contribution of the federal government and the Government of Quebec to municipal infrastructure funding through the MRIF. They measure the applicants’ project funding.
The variables that serve to measure the contribution of the federal government and the Government of Quebec to municipal infrastructure funding through the MRIF are as follows:
- The list and the amounts of submitted projects
- The list of projects funded under the MRIF
- The annual amount (2005 to 2011) of funding paid out to each municipality under the program
The impact measurement variables (dependent variables) that were used to analyze the impact of the MRIF program are as follows:
- Municipalities’ investment spending between 2005 and 2011
- Number of kilometres of drinking water pipes
- Number of ruptured drinking water pipes
- Number of instances per year in which bacteriological standards are exceeded
- Number of instances per year in which turbidity standards are exceeded
- Annual trihalomethane levels in the systems
- Number of kilometres of sewer pipes
- Score out of 100 for compliance with wastewater discharge standards
- Score out of 100 for compliance with overflow discharge standards
The control variables are as follows:
- Municipal infrastructure funding transfers under other joint Canada-Quebec programs between 2005 and 2011 (Gas Tax, Building Canada, etc)
- Transfers under other Quebec programs between 2005 and 2011
- Population (last two Censuses)
- Population under 18 (last two Censuses)
- Population aged 65 and over (last two Censuses)
- Median income and average household income (last two Censuses)
- Standardized property value from 2005 to 2011
- Administrative region
- Number of municipal employees from 2005 to 2011
- Total area
Appendix III: Logic model
MRIF 1: Drinking water and wastewater infrastructure
Construction, renovation or upgrading of
- drinking water supply infrastructure;
- drinking water treatment infrastructure;
- wastewater collection, interception and treatment infrastructure, including sanitary and combined sewers; and
- storm sewers
MRIF 2: Development of knowledge regarding municipal infrastructure
- Diagnosis of existing municipal infrastructure
- Assessment of infrastructure modernization requirements
- Identification of the most appropriate methods for modernizing the infrastructure
- Needs assessment for work aimed at ensuring the longevity of the infrastructure
MRIF 3: Infrastructure that supports local or regional development
Construction, renovation or upgrading of
- service infrastructure that supports the maintenance or development of commercial, industrial or tourist activities;
- cultural or artistic infrastructure that helps preserve, develop and promote culture and heritage;
- community recreational and sports infrastructure aimed at encouraging Quebecers to make sports and physical activity a part of their daily lives; and
- service infrastructure that improves citizens’ quality of life.
- Upgrading of drinking water systems
- Solution found to a lack of drinking water
- Environmental contamination issues are resolved
- Compliance with requirements in the case of existing wastewater treatment facilities
- Improved public safety
- Rehabilitation of pipes and aging infrastructure
- Knowledge of infrastructure condition and needs
- Establishment of priorities for action
- Development of planning and information tools
- Businesses added or consolidated
- Services for the public added or consolidated
- Direct long-term jobs created or maintained
- Tourism/cultural/heritage centres created or consolidated
- Recreational and sports infrastructure available to the public
- Communities use the infrastructure
- Improved drinking water quality and use
- Reduction in the negative effects of effluent on drinking water sources and aquatic ecosystems
- Improved efficiency of wastewater and storm water collection and treatment systems
- Long-lasting, reliable infrastructure tailored to needs (at a lower cost)
- Maintenance and development of community, commercial, industrial or tourism activities
- Preservation, development and promotion of culture and heritage
- Higher proportion of Quebecers who make sports and physical activity a part of their daily lives
Appendix IV: Survey summary tables
Readers should note that, in the tables in this document, the percentages are calculated on the basis of rounded-off numbers which may not necessarily add up to 100. A maximum variation of +/- 1% may be observed.
|Drinking water distribution system||Wastewater collection system||Drinking water treatment system||Wastewater treatment system|
|(2) Remained stable||19%||24%||10%||43%|
|(4) Does not apply||19%||19%||57%||29%|
|(4) Don’t know||2||2||2||2|
Source: Advanis Jolicoeur, 2012. Would you say the overall condition of the following infrastructure components has improved, remained stable or deteriorated over the past five years?
|Drinking water infrastructure||Development of knowledge regarding municipal infrastructure||Recreational, community or economic infrastructure, etc|
Source: Advanis Jolicoeur, 2012. Have you received funding under another infrastructure program since 2005 for the following work?
|(1) Through MAMROT||75%|
|(3) UMQ or FMQ||13%|
Source: Advanis Jolicoeur, 2012. How did you learn about the existence of the program?
|(1) No eligible work||35%|
|(2) Used a different program||25%|
|(3) Lack of time or knowledge||15%|
|(4) Non-compliant project||10%|
|(5) No projects at this time||5%|
|(6) Program has too many applications||5%|
Source: Advanis Jolicoeur, 2012. What is your main reason for not submitting an application under the MRIF?
|(1) Very easy||9%|
|(2) Quite easy||69%|
|(3) Quite difficult||16%|
|(4) Very difficult||6%|
Source: Advanis Jolicoeur, 2012. In general, would you say the project monitoring required by MAMROT was very easy, quite easy, quite difficult or very difficult for your municipality/organization?
|A) Obtaining project approval from the public or the municipal council||B) Completing the MAMROT funding application||C) Providing your share of the fundings||D) Obtaining the required (provincial or federal ) environmental authorizations||E) Issuing tenders for professional services||F) Issuing tenders for subcontracting work||G) Monitoring work progress||H) Conducting financial monitoring and submitting expense claims to MAMROT||I) Onsite audit by MAMROT of the project being funded|
Source: Advanis Jolicoeur, 2012. Indicate whether or not you experienced difficulties during each of the following phases.
|(1) Obtaining project approval from the public or the council||5%|
|(2) Completing the funding application||46%|
|(3) Providing your share of the funding||3%|
|(4) Obtaining environmental authorizations||2%|
|(5) Issuing tenders for professional services||2%|
|(6) Issuing tenders for subcontracting work||1%|
|(7) Monitoring work progress||8%|
|(8) Financial monitoring of expenditures for MAMROT||25%|
|(9) Onsite audit by MAMROT of the project being funded||7%|
Source: Advanis Jolicoeur, 2012. During which phase of your project did you receive the most support form MAMROT?
- Date modified: