Comparative study technology incubators in Quebec and abroad

2. Methodology used

The method used for achieving the objectives of the task assigned involves 14 broad activities, divided into four phases. It is illustrated in Figure 2.1. The main components of this process are set out below.

Figure 2.1 – Flow chart showing method used

In this document, the incubators' clients are referred to as "projects" since, depending on the organizations or phases of incubation, the clients may be firms or individuals. In all cases, the incubators' objective is for the projects to grow into thriving enterprises.

2.1 Planning the evaluation and collecting secondary data

First, a startup meeting was used to confirm the objectives of the study and examine the main methodological issues it involved. This also allowed the Agency to provide the Consultant with the secondary data it had available. After the meeting the work plan was prepared, with a timetable for doing the work within the timeframe established at the initial meeting.

An exhaustive list of the information needed for the study was then drawn up, after analysing each of the evaluation questions set out in Schedule A to the call for tender document and the methods of answering them. The list was prepared in such a way that each piece of information to be obtained would be connected with one or more of the evaluation questions (no pointless information) and each evaluation question would be connected with at least one piece of information (answers to all questions).

Once the list was complete, the secondary data sources were reviewed and all relevant data were assembled and evaluated. By doing this initial work of collecting and reviewing the secondary data we were able to determine what data were missing, among other things. As well, the work provided guidance for developing the collection tools, primarily the interview guide, to be sure that all of the primary data needed to complete the evaluation were obtained. It also provided the basis for a summary characterization of the various technology incubators, to be used in selecting the sample for the study.

The initial data collection was followed by a meeting with the people in charge of Inno-centre. The meeting gave the Consultant an opportunity to acquire more detailed knowledge of the organization, become more familiar with the problems associated with technology incubators in general and validate the understanding of the secondary data collected, and provided guidance for developing tools for collecting primary data.

The tools for collecting and analysing the data – the background information sheets for describing the incubators selected, the interview guide for the telephone survey and the survey questionnaire to use with the incubated enterprises – were then prepared. First, the background information sheets and list of information needed were submitted to the Agency for approval. The interview guide and survey questionnaire were then prepared, so that data could be obtained in relation to all of the items on the list.

At the same time, information obtained from the secondary data collected was used to select a sample of incubators to be used for the benchmarking study. In addition to Inno-centre, the sample included seven other incubators funded by the Agency and 20 incubators outside Quebec located elsewhere in Canada or in other countries that had business models similar to Inno-centre's or otherwise relevant to the study. While some elements of the business model were taken into account in selecting the sample, the most important factor was to ensure that the incubators selected offered the same type of incubation services to enterprises that were at the same startup stage. It was therefore necessary to avoid selecting incubators that offered earlier pre-incubation services, and later support services for enterprises that were already well established.

Phase 1 was then concluded with the production of the methodological report.

2.2 Collection of the primary data

Two methods were used to collect the primary data. First, telephone interviews were conducted, using the guide prepared in Phase 1, with managers of the other Quebec incubators funded by the Agency and of the Canadian and international incubators selected. The people in charge of Inno-centre were also contacted, to supplement the information obtained at the initial meeting. Second, the survey questionnaire that was also prepared in Phase 1 was administered by email to the firms that were being or had been incubated in the various incubators being examined.

The telephone interviews and survey of firms followed the process set out in Figure 2.2.

Figure 2.2 – Detailed outline of the collection of primary data

2.2.1 Telephone Interviews

For the telephone interviews, a key interlocutor was first identified for each incubator. In general, but not always, this was the senior manager of the incubator. A letter signed by the Agency was then sent to that interlocutor, describing the study and introducing the Consultant. An initial contact was then made by telephone or email, to explain the study, answer any questions the managers had and make an appointment for the telephone interview. After that, and before the date of the appointment, the interview guide was provided for the incubator representatives so they could prepare for the interview and assemble the data needed. After this, the telephone interviews were conducted.

In Quebec, the interviews were conducted between February 14 and March 7, 2008. Of the eight incubators studied (including Inno-centre), one could not participate in the interviews, but sent back its completed interview guide. The eight Quebec incubators that participated in the study were:

• Inno-centre (Montreal);
• Bionov (associated with the Centre de développement des biotechnologies de Sherbrooke);
• Centre de développement d’entreprises technologiques ( CDET - Gatineau);
• Ag-Bio Centre (Lévis);
• Centre incubateur d’entreprises de la Montérégie ( CIDEM - Brossard);
• Centre québécois d’innovation en biotechnologie ( CQIB - Laval);
• Centre d’entreprises et d’innovation de Montréal ( CEIM - Montreal);
• Centre de l’entrepreneurship technologique (CENTECH - Montreal).

Outside Quebec, the interviews were held between February 18 and 29, 2008. Of the 20 incubators contacted, five never replied to telephone calls or emails, three refused to participate in the study,¹ three were prepared to participate but did not have the time needed for it within the necessary timeframe and nine answered the survey questions. Of the nine, one was in Ontario, one in the United States, one in Belgium, one in Scotland, one in Israel and four in France. The nine incubators outside Quebec that participated in the study were:

• MaRS (Ontario);
• University of Central Florida Technology Incubator (United States);
• Wallonia Space Logistics (Belgium);
• Hillington Park Innovation Center (Scotland);
• Misgav (Israel);
• Incubateur Midi-Pyrénées (France);
• Incubateur régional d’Aquitaine (France);
• Enterprise Innovation Centre de Montpellier-Agglomération (France);
• Agoranov (France).

2.2.2 Client Survey

The survey of the incubated enterprises was conducted as follows. First, the survey population was established. It consisted of all enterprises still in operation that had an email address and were then or had been clients of one of the incubators being examined. The list of enterprises in the survey population was prepared using information available on the Internet and the lists of enterprises supplied by the Agency. Accordingly, only enterprises that provide an email address on the Internet were included in the survey population. Second, the survey questionnaire was sent by email to the enterprises to be surveyed, together with a letter signed by the Agency describing the study and introducing the Consultant. Third, to maximize the response rate, two reminders were sent out after the questionnaire was sent.

In total, 950 emails were sent out in the course of the survey of incubated enterprises: 283 to client enterprises of the Quebec incubators and the remaining 667 to client enterprises of the incubators outside Quebec. However, 109 of those emails were undeliverable (101) or sent to enterprises that had not been incubated (8). The real survey population was therefore 841 enterprises. Ultimately, 45 enterprises replied (22 in Quebec, 23 outside Quebec), giving a response rate of 5.4%.

Once the primary data collection was complete, the information gathered in respect of each incubator in the sample and the incubated enterprises was compiled using an analytical table to assist in evaluating and comparing the data.

2.3 Analysis of Results and Preparation of Preliminary Report

This phase consisted of analysing and interpreting the information collected in the earlier phases, and then producing the preliminary report on the study. The work was subdivided under the following four headings:

• a descriptive analysis of the incubators' business models;
• a descriptive analysis of the incubators' performance;
• a comparative analysis of the various incubators' business models and performance;
• a supplementary analysis to provide answers to the evaluation questions.

2.3.1 Descriptive Analysis of the Business models

The first component of the analysis consisted of a detailed description of the business models of the incubators in Quebec and elsewhere that were being examined. An incubator's business model can be characterized based on multiple dimensions covering various aspects of its operations and its relationship with its environment. For the purposes of this study, the following dimensions were used to characterize the business models:

• basic characteristics (mission, status, type of facility, history and economic sectors covered);
• composition of the team and the network;
• client selection process;
• services provided;
• nature of the coaching provided, including strategies for finding funding;
• follow-up with graduates;
• the incubator's business development;
• source of the incubator's clients;
• efforts allocated to the various activities;
• the incubator's financial structure (expenses and revenues), including its payment methods.

The business models were described by dividing the incubators into two groups: those in Quebec and those outside Quebec.

The results of the descriptive analysis of the business models are set out in Chapter 3 of this report.

2.3.2 Descriptive Analysis of the Performance of the Incubators

The second component of the analysis consisted of a description of the incubators' performance. There are two dimensions to an incubator's performance. The first involves the incubators' actual results: essentially, the number of projects undertaken and completed. The second dimension relates to the results of the enterprises that are being incubated or have graduated. The following quantitative indicators were used to describe these two dimensions of the incubators' performance.

Incubators' results:

• operating budget;
• number of projects evaluated per year;
• number of projects accepted for incubation per year;
• number of projects in incubation per year;
• number of graduates per year.

Enterprises' results:

• survival rate for enterprises five years after beginning incubation;
• total number of jobs;
• total value of sales;
• percentage of enterprises that obtained funding;
• total value of funding obtained.

These are referred to as primary indicators, in that they consist of gross data that has not been manipulated, other than by calculating yearly averages for the last five years. As well, to facilitate comparisons, all amounts shown are in Canadian dollars. To do this, the figures supplied by the managers of international incubators in Euros, pounds or U.S. dollars were converted into Canadian dollars using the annual average exchange rates from the Bank of Canada.² However, so as not to skew the analysis, the amounts were left in their original currency form in doing the intermediate calculations.

The indicators of the enterprises' results were measured separately for enterprises in incubation and graduates. However, because of the scarce data, the total value of sales for graduates was not compiled.

Table 2.1 shows the number of incubators inside and outside Quebec that provided data, for each of the primary quantitative indicators.

Table 2.1 - Number of incubators that provided data for each of the quantitative indicators

Indicators	Incubators in Quebec (n = 8)	Incubators outside Quebec (n = 9)
Incubators' results
Operating budget	8	9
Number of projects evaluated	6	5
Number of projects accepted for incubation	8	9
Number of projects in incubation	8	9
Number of graduates	8	9
Enterprises' results
Five-year survival rate	4	5
Number of jobs in enterprises in incubation	6	4
Sales by enterprises in incubation	3	4
Percentage of enterprises in incubation that obtained funding	3	3
Funding obtained by enterprises in incubation	3	3
Number of jobs in enterprises that graduated	4	5
Percentage of enterprises that graduated that obtained funding	3	2
Funding obtained by enterprises that graduated	3	3

In addition, some of the indicators set out above cannot be used directly to compare the incubators. The five-year survival rate for enterprises and the percentage of projects that obtained funding are the only indicators that can be used directly; the others depend in part on the size of the incubator. It was therefore necessary to create secondary indicators, using these primary indicators, to take the size of the incubators into account in the comparative analysis.

The three measurements used for the size of the incubators were average operating budget for the last five years, annual average number of projects in incubation, and an estimate of the total number of graduates. That estimate was obtained by multiplying the annual number of graduates for the last five years by the number of years that each incubator had been in existence.

Based on these size measurements and the primary indicators, the following 14 secondary indicators were constructed:

1. Admission rate = Number of projects accepted/Number of projects evaluated.
2. Cost per enterprise incubated = Operating budget/Number of projects in incubation.
3. Cost per graduate = Operating budget/Number of graduates per year.
4. Graduates per enterprise incubated = Number of graduates per year/Number of projects in incubation.
5. Jobs per enterprise incubated = Number of jobs in enterprises in incubation/Number of projects in incubation.
6. Cost per job (enterprises incubated) = Operating budget/Number of jobs in enterprises in incubation.
7. Sales per enterprise incubated = Value of sales by enterprises in incubation/Number of projects in incubation.
8. Lever effect – Sales (enterprises incubated) = Value of sales by enterprises in incubation/Operating budget.
9. Funding per enterprise incubated = Funding obtained by enterprises in incubation/Number of projects in incubation.
10. Lever effect – Funding (enterprises incubated) = Funding obtained by enterprises in incubation/Operating budget.
11. Jobs per graduate = Number of jobs in enterprises graduated/Estimate of total number of graduates.
12. Cost per job (graduates) = Operating budget/Number of jobs in enterprises graduated.
13. Funding per graduate = Funding obtained by enterprises graduated/Estimate of total number of graduates.
14. Lever effect – Funding (graduates) = Funding obtained by enterprises graduated/Operating budget.

The results of the descriptive analysis of performance are set out in Chapter 4 of this report.

2.3.3 Comparative Analysis of Incubators

To identify best practices in technology incubation, the existing links between the various components of the business models of the incubators inside and outside Quebec, and the performance of those models, must be determined.

Accordingly, an analysis was done to compare the incubators based on a number of criteria that describe their performance and the various dimensions of their business models. The incubators were put in order by rank for each criterion. An incubator was classified as "High" if it ranked among the best three (dark), "Average" if it ranked among the middle three (pale) and "Weak" if it ranked among the bottom three (white).³

The performance of the incubators inside and outside Quebec was evaluated using the following four broad criteria, which were defined and measured using the secondary indicators described above:

The cost of incubation simply corresponds to the cost per incubated project. For this criterion, an incubator that performed better will have a lower cost per enterprise incubated.

The productivity of the incubators is a combination of the cost per graduate and the number of graduates per incubated project. This criterion was measured using a composite consisting of an average of the incubators' rankings for the two indicators considered. They were then classified using the average ranking produced by that composite index. For this criterion, an incubator that performed better will have a lower cost per graduate and a higher number of graduates per incubated project.

The survival rate corresponds to the proportion of enterprises still in operation five years after their incubation began. For this criterion, an incubator that performed better will have a higher survival rate.

Client enterprises' performance was measured using different indicators of the results of the incubated enterprises and graduates: cost per job, jobs per incubated project/graduate, lever effect on sales, sales per incubated project/graduate, percentage of enterprises that obtained funding, lever effect on funding and funding per incubated project/graduate. Each incubator was ranked using a composite index constructed using the following methodology:

• First, for each incubator included, the incubators were each compared to the others. For each pair compared, the incubator with the best performance was assigned one point, and the total of the points resulting from all possible comparisons was determined for each incubator. This step was used to identify the best incubators for each indicator.
• Second, for each incubator included, the total points obtained by the various incubators was divided by the number of incubators for which data was available for that indicator. This step was used to weight the various indicators based on the availability of data. A particular number of points would be worth more where the points were for an indicator where it was possible to compare more incubators.
• Third, the total of the ratios was calculated for each incubator. This step was used to group incubators' performance based on the various indicators.
• Fourth, the final composite indicator was obtained by calculating the average of the ratios, by dividing the total of the ratios for each of the incubators by the number of indicators for which each one had supplied data.
• Fifth, top ranking was assigned to the incubator that had the highest composite indicator, that is, the one that compared most favourably to the others over all indicators of the client enterprises' results, and so on.

These performance criteria were then compared with the criteria describing the business models. This analysis was done using the dimensions of the business models identified earlier: general characteristics, team and network, selection process, services offered, coaching, allocation of efforts and financial structure. The various criteria used in analysing each dimension of the business model, and the factors that guided the classification, are explained below.

To protect confidentiality, the comparative analysis does not name the incubators.

The results of the comparative analysis are set out in Chapter 5 of this report.

2.3.4 Supplementary Analysis

The three analyses discussed above provided answers to some of the main evaluation questions, but not all of them. A supplementary analysis was therefore needed, using some of the preceding analyses, but, most importantly, examining other aspects of technology incubation that were addressed in the interviews or in collecting the secondary data.

The results of the supplementary analysis are set out in Chapter 6 of this report.

2.4 Distribution of Final Report

The study concluded with the production of the final report, which includes the client's comments on the preliminary report and submission of the report to the project managers.

2.5 Methodological Limitations

The effects of the methodology used to achieve the objectives of the study, and the technical constraints associated with it, mean that the conclusions from the study are limited in several regards. The main methodological limitations that must be taken into account are as follows.

• Because of the small size of the sample of incubators outside Quebec, the results of the study must not be regarded as representative of incubation in general; rather, they are a reflection of what may be done outside Quebec by incubators that offer incubation services for enterprises at the startup stage in technology-related economic sectors. As well, the nine incubators outside Quebec examined in this study are the ones that, out of the 20 contacted, agreed to respond.
• At the same time, this study compares an almost exhaustive sample of incubators in Quebec with the sample of incubators outside Quebec. From that perspective, caution must be exercised in generalizing the conclusions that may be derived from the comparison.
• Because of the major differences between the incubators' structures and operating methods, even though they were selected because of their similarities in certain regards, and because of the limited number of incubators compared, the averages calculated for a majority of variables involve large variances, and caution must therefore be exercised in interpreting them.
• Despite the fact that more intensive effort was invested than had been planned, particularly in sending out additional reminders, the survey of the incubators' clients produced only a low response rate. Accordingly, the results of that survey were considered only summarily, and solely as support for the results of the interviews.
• It was not possible to obtain all of the quantitative data requested at the interviews (financial statements, results of incubators and results of incubated projects and graduates), essentially because the incubators do not all evaluate their performance and the clients' performance in the same way; as a result, they do not all collect the same information. A number of variables therefore involve data that may be fragmentary for some years or some incubators, or both. This situation complicated the analysis and limited its scope.
• It was virtually impossible, to start with, to determine the number of patents obtained or licences granted by the incubators' clients over the last five years, and so these items are not considered here.
• To make up for the fragmentary nature of the data, particularly in respect of the incubators' financial structure and performance, different indicators were grouped into composite indexes. Caution must therefore be exercised in interpreting the results of the study.
• The data that are most fragmentary relate to the results of the graduate enterprises, for three main reasons. First, not all of the incubators define graduates the same way. Second, the incubators that compile data on their graduates do not all use the same methods for that purpose. For example, one incubator collects data over five years, and then stops following up and the statistics are recorded as they stand in the fifth year. Another compiles data only when it has a share in the enterprise. These methodological differences make comparisons difficult. Third, other incubators simply do not compile data on their graduates.
• At the same time, the incubators' various operating methods can introduce a degree of bias into the data. For example, the international incubator for which 100% of incubated enterprises and graduates obtained funding achieves that result because it pays all of its clients a refundable advance when they start the incubation.
• The monetary data have been converted into Canadian dollars, applying the exchange rate for the year of the figure in question. However, the comparisons made in the study do not take into account cost of living differences between countries, such as rent. The differences observed may therefore be wider in reality than the data show.