Accelerating into Control

Researchers have generally found that entrepreneurs are more optimistic and more confident than non-entrepreneurs. While it may help entrepreneurs persevere in the face of potential business failure, we cannot mistake their confidence for always knowing what to do with their business idea. Entrepreneurs in fact seek out mentors and other useful connections to help them succeed throughout the growth of their businesses, particularly at the start. Many entrepreneurs seek advice informally and in a piecemeal manner, but some seek more formal assistance through structured or semi-structured entrepreneurship programs. Indeed, we currently are witnessing the rise of the “support ecosystem,” which offers a plethora of entrepreneurship education and training programs. These programs vary in their design and operation; some, for example, are run by universities and colleges, some are offered by nonprofits or the government, and others are offered by for-profit entities. They might operate just a weekend in length, or last several months or years. The scope of a program’s intervention and how closely it works with each entrepreneur or startup varies widely. With this increase in the number and scope of program offerings, we wonder if adoption is outpacing evidence of their effectiveness. In this article, we examine various types of programs, with a primary focus on the accelerator, provide some context for current research and research concepts in this area, and discuss some implications of collecting data for program operators and policymakers.

innovations / Accelerating Entrepreneurship entrepreneurship programs as "bad, " or conversely, promote them as "good. " It means at present we are at best ambiguous about the overall effects accelerators have on entrepreneurs. To help entrepreneurs, program operators, and funders of programs, we therefore call for studies that examine what accelerators do well and what they don't do well.
Finally, we review the concepts of treatment and control groups and randomization. One of our primary arguments is that control and treatment groups are necessary to explain the counterfactual-that is, what would have happened to an entrepreneur/startup if it had not participated in a program-and that they are overlooked by both researchers and practitioners. With any given entrepreneurship education program, there are individuals who receive the treatment of the program, and there are individuals who do not receive it. Following these two groups enable researchers to compare results and establish a counterfactual outcome.
How the control and treatment groups are constructed determines the strength of any findings. Consider the people an entrepreneurship program attracts. The decision to enter a program not only identifies someone as an entrepreneur, but as an entrepreneur who seeks formal assistance through a program, which is a distinct subset of entrepreneurs. This choice of actively seeking assistance could be associated with a number of factors. For example, we might imagine that someone who feels they could benefit from structured assistance in the startup process has greater potential than someone who lacks such foresight and planning skills. Conversely, the entrepreneur who needs structured assistance could have less potential than someone who perseveres without it. Therefore, this treatment group cannot be compared to just any individual or group of entrepreneurs, as the control group must have characteristics that are highly similar to the treatment group, including the desire to enter a program.
Compounding the issue is that many programs, accelerators in particular, do not accept all applicants. An accelerator may receive hundreds or even thousands of applications for a handful of openings, and may also actively recruit entrepreneurs/startups rather than accept only previously unknown applicants. This means that applicants are vetted and vetted again, until only the select few are accepted into the program. This vetting process could play a significant role in determining outcomes regardless of the actual components of the program.
This bring us to the significance of randomization, or the process of randomly assigning treatment. By randomly selecting who receives treatment from a pool of potential program participants, the control and treatment groups can be assembled such that differences in their outcomes are attributable to the program. Without randomizing, it is difficult to confidently identify the specific effects of the program. We realize such random selection is unlikely to occur in the real world of accelerators, whose modus operandi is picking winners and accelerating their growth cycles, not taking in any random interested entrepreneur or startup. We will discuss alternatives later in this article, but random sampling remains the desired research design. 7 From a research perspective, we do not know much about the effectiveness of entrepreneurship education and training programs, let alone accelerators specifically. A recent special issue of the Journal of Small Business Management spoke to innovations / volume 8, number 3/4 141 innovations / Accelerating Entrepreneurship this problem. 8 It featured some analysis of these programs, but the focus was more on measures of individual competencies and skills and entrepreneurial intent than actual entrepreneurship outcomes (see Table 1). Some of the papers in the special issue spoke to the fundamental concern about control groups, but most did not, as demonstrated by a key phrase search, presented in Table 2.
Frankly, a discussion of entrepreneurship education evaluation without mention of control groups is missing the point.
This is not to say that there is a total lack of research on outcomes or of research designs that include control and treatment groups. However, this work is mostly limited to international studies in developing nations, where programs are much less costly to run, and the ability to extract lessons for developed nations is limited. 9 Jared Konczal

Accelerating into Control
One such comprehensive effort is a forthcoming paper assessing 60 selected entrepreneurship education and training evaluations. It groups evaluations into four different areas, based on their target audiences-secondary education students (9), higher education students (10), potential entrepreneurs (16), and practicing entrepreneurs (25)-and three tiers of evaluation rigor. 10 Tier 1 studies (21) are randomized controlled experiments with an experimental design. Tier 2 evaluations (10) follow a quasi-experimental design, with some control and comparison groups that are not purely randomly assigned. Tier 3 evaluations (29) consist of surveys of program participants, including tracer studies, and monitoring and evaluation reports that rely largely on administrative data.
Of these 60 studies, 19 concern programs that operate in what the International Monetary Fund defines as advanced economies. 11 Eight are located in the United States: Tier 1 (1): The Growing America Through Entrepreneurship Project (GATE) Experiment evaluation, which is the only randomized control trial conducted on entrepreneurship education within the United States that we are aware of (not job training or programs marketed for self-employment/the unemployed) 12 Tier 2 (1): A high school program in Boston run by the Network for Teaching Entrepreneurship Tier 3 (6): Two university entrepreneurship programs, one national program for small employers in disadvantaged communities, two State programs for women entrepreneurs, and one county program for minority entrepreneurs Eleven are in other advanced economies: Tier 1 (1): The Bizworld program in the Netherlands for primary school children Tier 2 (4): Two programs for secondary school students, one for university students, and one that teaches strategic plan management to dairy farmers in the Netherlands Tier 3 (6): One Danish program for secondary school students, three programs for university students, and one small program for executives in Northern Ireland Within the Tier 1 and Tier 2 programs in these advanced economies, only the GATE experiment stands out as having relevance for accelerators. The remaining 41 studies were conducted in non-advanced economies, with 19, 5, and 17 studies in Tier 1, 2, and 3, respectively.
The GATE program was offered free of cost through 14 small business development centers and community organizations across seven cities in three states, mostly in the northeastern United States. It was marketed to anyone interested in starting or growing a business and generated 4,197 complete applications. Treatment and control groups were randomly assigned through a lottery, with the non-accepted applicants serving as the control. Participants first received an individual assessment, followed by training in classrooms and/or one-on-one settings provided by experienced business consultants. The overall finding was that program participants were more likely than non-participants to be business owners six innovations / volume 8, number 3/4 143 144 innovations / Accelerating Entrepreneurship months after the time of application, but the differences between the treatment and control groups diminished significantly when measured after one year and five years. The study also documented that many of the entrepreneurs not selected for the intervention went on to receive training from some other program. 13 This provides some helpful reference points for accelerators; for example, that outcomes should be tracked longitudinally to capture short-term and long-term effects, and the realization that an entrepreneur/startup that doesn't make it into one accelerator might enroll in another program, thereby segmenting the control group sample.
With the lack of published research and only minimal outcomes data provided by programs, what are program operators doing? What is currently considered "good" information about entrepreneurship programs is summary statistics about entrepreneurs and startups that go through these programs. Aggregator websites and accelerator organizations, such as f6s.com and gan.co, have sparse information on program outcomes. In an environment where phrases like "value proposition" and "value-added" are used like commas, it is frankly surprising that there is such a dearth of detailed data that demonstrates the real value of programs or provides the level of detail needed for serious analysis by individual companies. We should at least strive for alternative measures like revenue and employee growth of companies over time, though we propose something that takes outcomes measurement a few steps further.
It is interesting to read that, of 25 startups in a program, 23 received funding (and maybe we are given the dollar amount), 22 are still operating, and 15 have already hired employees (and maybe we are given the number of employees), but this information fails to address the effect of the program itself. It does not tell us about what would have happened to the entrepreneur or startup absent the program. These statistics mix program inputs and outputs, and funding from investors does not guarantee the success of the business. 14 Moreover, if a program merely declares one of their startups a success, it is hard to know whether that startup would have been just as successful, less successful, or perhaps even more successful without the intervention. We need to remedy this lack of counterfactuals.

HOW DO WE GET THERE?
There are understandable limitations to the research concepts we have discussed. Counterfactuals rely on control groups, which represent an added cost and require a larger population to be tracked and monitored. Ideally, randomization would be involved when building control and treatment groups, though we recognize that this is impractical for accelerators. This type of study is also particularly susceptible to falling apart during the longer time periods needed to observe outcomes. For example, one evaluation of a Mexican government program shut down because funding dried up. 15 Accelerators have a number of operational factors that might seem prohibitive to researchers or even allowing the programs themselves to track outcomes in this way. While the initial number of applicants can be quite high, the number accepted is typically quite low, which provides a limited sample of treated entrepreneurs/startups. A study on a small sample of Indian textile firms that received subsidized management consulting got around the small sample size by using measures with higher frequency. Researchers observed weekly machine data about product quality and measured the implementation of recommended practices monthly. 16 A similar concept may be feasible for entrepreneurs/startups within an accelerator, but not for the control group. A possible solution to small treatment and control groups would be to group a large number of accepted entrepreneurs/startups together by combining multiple cohorts from one accelerator or pooling data on startups across multiple accelerators. Besides the effort needed to coordinate such partnerships, there are other issues with this approach. While the foundation of intervention-mentoring and structured educational componentsis the same across groups, individual treatment will vary across accelerators, not only financially but programmatically. Accelerators may target different stages of entrepreneurs/startups, from the pre-launch idea stage to a company that is growing and generating revenue, thereby limiting comparability. The treatment of different cohorts will differ even within one accelerator, as they tweak their program structure along the way. Nevertheless, accelerators offer some pluses for researchers and data analysis. While various programs might have different treatments, their general focus on Internet and software startups provides some uniformity in the type of entrepreneurs and companies to be studied. Moreover, the relatively short intervention period-typically from six weeks to six months-means that new cohorts are generated relatively frequently, thereby increasing sample size for statistical analysis. Perhaps one of the most significant advantages for researchers is that accelerators interact with entrepreneurs. Much of the research on entrepreneurship relies on approximations of entrepreneurial activity, such as self-employment and business ownership, which are not uniformly agreed upon as entrepreneurship. The debate will likely never be settled, but we argue that individuals engaged with accelerators are close to whatever murky consensus there might be about the definition of an entrepreneur. 17 Above all, we should remain committed to understand the foremost purpose of accelerators. If the primary goal of the accelerator program is to pick good candidates and advance companies to a level they would have reached without the intervention, only more quickly, how can the program be assessed conclusively without a comparison group? Moreover, if randomization is impractical, how can we build a control group? The next best option relies on a quasi-experimental design that builds a control group out of those who just missed the cut for the program. It seems reasonable to think that the dozen next best are nearly similar to those who were selected. In the case of accelerators in particular, we are not convinced that other methods for building control groups, such as propensity score matching, would appropriately control for the selection effect. Three primary innovations / volume 8, number 3/4 145 innovations / Accelerating Entrepreneurship issues present with this method: deciding whom to follow, the entrepreneur or the company; deciding what information to collect; and undertaking the actual task of following companies and collecting data. The composition of startup teams is fickle and apt to change. 18 Thus we face the fundamental question of what to do when founders leave a startup or if a company exits and there is no longer a firm to follow. Because an accelerator's intervention revolves around an idea or company and not individual founders, the focus should remain on the idea or company, and changes in team composition should be noted in data collection. This is not an exhaustive list, but it is representative of inputs and outcomes accelerators and researchers should seek to know about companies in their programs: basic demographic information of founders (age, race, gender, marital status, educational attainment), founder's prior exposure to entrepreneurship, operating status (and if exited, what kind of exit), revenue status (and what amount generated), number of employees, whether the company is profitable, whether it has received additional financial capital (not just additional venture funding, but all sources), and whether the founding team members are still with the company (and if not, their current employment status). There are also some additional points of information needed for programs that accept entrepreneurs at the idea stage. Mainly, if working with ideas, we need to establish whether the idea turned into a legally incorporated company. Non-accepted startups/ideas should be asked similar questions about their demographics and company outcomes, and whether they ultimately enrolled in another accelerator or entrepreneurship training and education program.
To collect data, accelerators and researchers might begin by scraping together data from third-party networks, such as LinkedIn, to follow the employment status of founders and passively collect information about the startup. Many platforms already either require or strongly encourage signing in with e.g. a LinkedIn or Facebook account to fill out an accelerator application. As these social networking sites improve and further develop their application program interfaces, the list of applicants with matched social network accounts could prove useful. Accelerators could also administer survey questionnaires upon program completion, six months after, and annually thereafter to follow outcomes longitudinally. Those accepted by the accelerator should be relatively easy to follow, as they will have strong ties to the program operators, not only on a personal level but in many cases financially. Following the group that is not accepted is more problematic, but if the application process includes the expectation that all companies will be followed, the accelerator will be able to follow non-participants much more easily. For example, response rates in the GATE experiment were lesser in the control group, but not prohibitively so. 19 Accelerators that use a two-stage selection process (initial application plus calls for follow-up information from the first cut of companies) might remind applicants of the expectation during the secondary submission process.
We can envision that, as multiple cohorts run through an accelerator program, the ongoing data collection will involve different stages of companies all at the same time. Ultimately, the result will be a dataset, perhaps across multiple accelerators, on entrepreneurs/startups that have gone through the accelerator, and data on similar entrepreneurs/startups that were not accepted. We argue that comparisons between these two groups provide a much better basis for talking about the effects of these programs than current practice.
We turn now to policymakers and government funding. With the increased number of accelerators, there is more competition not only for attracting good companies but for funding. As accelerators have risen to prominence, it is tempting for government officials and other economic development agencies to turn to them as a way to spur economic growth, whether by directly investing in or sponsoring an accelerator, or by subsidizing its operation with tax expenditures. The purpose of this paper is not to argue whether or not governments should involve themselves in this activity, though policymakers might be wise to heed lessons from history and approach public-private partnerships carefully, 20 and we would make no assertion other than that these programs are empirically unproven. Rather, our purpose is to drive the discussion on outcomes, on which public funders should rightfully condition their support. They should not seek information solely on economic impact outcomes, such as jobs and companies created or revenues generated, but also should consider the counterfactual. If necessary, to enable an accelerator to be studied, they should increase their support and provide more funding for data collection on non-selected companies. Whether a subsidy or direct investment in an accelerator is worthwhile should be judged not only on whether that money brings a return, either financially through an equity stake or through economic development, but on whether that money could have been better spent elsewhere. Entrepreneurs are also citizens and, as such, desire good amenities, good roads and infrastructure, good schools, and a good quality of life. Policies and programs for assisting the general public in many cases can be just as (if not more) effective as those that specifically target entrepreneurs. When considering whether to implement policies and support programs specifically targeted at entrepreneurs, including accelerators, policymakers should weigh whether that support would be better spent on broader public support that would indirectly but effectively also assist entrepreneurs, thereby promoting both economic development and overall public welfare.