Flaws and weaknesses of EIA as event evaluation methodology

Introduction

This essay serves to first explain the rationale for event policy and economic impact assessment (EIA) and then explore the event evaluation literature seeking critique concerning the adequacy, suitability or effectiveness of EIA as legitimation for event policy. In doing so it aims to identify the flaws and weaknesses inherent within traditional event evaluation techniques. A review of the literature will identify these flaws and weaknesses and note evidence of recent efforts to address them. Then a discussion of models, multipliers and their potential misapplications follows leading to a brief conclusion before which key data from three prominent academics are tabulated to summarise key themes, variables and factors warranting closer attention in event research, policy and practice.

Rationale for event policy

Globalisation and deindustrialisation have seen many governments come to recognize tourism and events as catalysts for economic development (Chalip & Leyns, 2002; Masterman, 2009; Getz, 2012). The rationale for public policy concerning the expenditure of public funds on events is that they foster numerous and notable, tangible and intangible socio-economic and socio-cultural benefits to the host community. Although there is much contention about the credibility of certain evaluation methods, these benefits and approbations are repeatedly cited throughout the tourism and events, public policy, and wider literature (e.g., Ritchie & Beliveau, 1974; Ritchie, 1984; Chalip, 2003; 2006; Masterman, 2009; Getz, 1997; 2012; Foley, McGillivray & McPherson, 2012).  Resultantly, event policy is becoming increasingly embedded into public policy objectives (Foley, McGillivray & McPherson, 2012; Dredge & Jenkins, 2007). Leading economist, Dr. Peter Abelson (2011, p.49) identifies four principal event policy objectives as, maximising; gross domestic product (GDP) or gross state product (GSP), net income or consumption of households within the destination, employment within destination and net welfare benefits to residents within the destination.

Rationale for EIA

The conceptual thinking that underlies the need for legitimation of investment of public funds in festivals, events and facilities can be contextualised as follows. Residents of a community contribute to public funds through taxation and rates and the local council uses amounts from this revenue to subsidise or sponsor production of a festival or an event, or to develop event infrastructure or amenities. These events and/or facilities attract visitors who are not residents of the funding community and ipso facto inject new money into the funding community (Crompton, 1995). This (new) money is purported to create income and jobs for the funding community and, therefore, represents return on investment. In the context of events, economic impact can be considered the take-home economic revenue of a host community resultant from spending directly attributed to a specific event or facility (Turco & Kelsey, 1992). The rationale for an EIA is therefore, to measure these economic benefits (Crompton, 1995) in order to legitimate the expenditure of public money (Foley, McGillivray & McPherson, 2012) and realise wider policy objectives (Abelson, 2011).

 As awareness and interest in events as an economic catalyst has increased, so have the number of related economic impact studies (Delpy & Ming, 1997). These studies are often commissioned by a variety of stakeholders including not only academics and researchers, but event organisers, sponsoring government and non-government organisations. EIA are sometimes undertaken for differing and potentially compromising objectives (Kumar & Hussain, 2014). The next section will examine some of the flaws and weaknesses of EIA identified in the literature.

Flaws and weaknesses of EIA

Reviewing the literature, there is no paucity of critique toward EIA as an effective instrument for event evaluation. Papers citing flaws and weaknesses of the methodologies, economic models and multipliers used for conducting EIA are abundant. For example; inappropriate use of impact multipliers (Archer, 1976; 1984; Wanhill, 1988; Crompton & McKay, 1994; Crompton, 1995; Harris, 1997; Chang, Kim & Petrovcikova, 2015; Andersson, Armbrecht & Lundberg, 2016), duplicity of economic models (Frechtling, 1994; Dwyer & Jago, 2015; Dwyer, Jago & Forsyth, 2016), data collection methods (Ritchie, 1984; Getz & Frisby, 1988;  Delpy & Ming, 1997; Breen, Bull & Walo, 2001; Kumar & Hussain,2014; Chang, Kim & Petrovcikova, 2015), lack of defined objective (Armbrecht & Anderson, 2016; Dwyer, Jago & Forsyth, 2016; Tindall, 2011), failure to factor opportunity costs, intangible costs, welfare and benefits (Dwyer & Jago, 2015), reliance on assumptions (Crompton, 1995) and the non-academic nature of many of the studies (Testa & Metter, 2017; Armbrecht & Andersson, 2016; Kumar & Hussain, 2014; Janeczko, Mules & Ritchie, 2002). These themes are prevalent throughout the event evaluation literature and, according to Delpy and Ming (1997, p. 230), it seems that the complexity of conducting economic impact research, the wide-ranging scope of event tourism, and the plethora of variables requiring consideration, has resulted in many EIA to be flawed in design and almost always result in exaggerated and inaccurate conclusions as to the economic impacts of events to a community.

Models, multipliers and misapplications: Muddles or misnomers?

An EIA requires the use of an economic model and appropriate multipliers. These models differ considerably in intricacies such as quality and accuracy of results, data requirements, and basic assumptions (Klijs, Heijman, Maris & Bryon, 2012). Models are, however, often selected without reflection of these differences (Crompton, 2006; Dwyer, Forsyth & Spurr, 2004). The economic models most commonly selected are input/output model (I-O) (Crompton, 2006) and, but to a lesser extent, computable general equilibrium (CGE) (Dwyer, Forsyth & Spurr, 2006). However, the frequency of cost/benefit analysis (CBA), which goes a step further than EIA in attempting to place a monetary value on environmental impacts (Pearce, Atkinson & Mourato, 2006), are increasing in recent years largely since I-O and CGE (and hence, EIA) have proven inadequate in data collection for evaluating the fourth policy objective identified by Abelson (2011); i.e. maximising net welfare benefits to residents within the destination. Although Blake (2005) notes that CGE can contain measures of welfare and Dixon (2002) rightly points out that some CGE are designed specifically to measure welfare, Dwyer, Forsyth and Spurr (2006), Dwyer and Jago (2015), and Dwyer, Jago and Forsyth (2016) are adamant that neither EIA nor CBA are alone adequate in methodology for holistic evaluation of events. These authors note the relative paucity of CBA case studies in the literature as opposed to the abundance of EIA and argue that CBA and EIA are mutually opposing anyway. They go further to state that EIA will always show a positive result whereas CBA often yields the opposite and that, since economic impacts are not the same as economic benefits, the two evaluation methods need to be somehow synthesised. Of note, is that Andersson, Armbrecht and Lundberg (2016) have made some real and recent progress in this direction through field testing a new model (triple impact assessment) for measuring impacts of a major sporting event (in this case, the 2013 European Athletics Indoor Championships). This evaluation was conducted totally from sustainability perspectives and factors diverse data in a common monetary metric. The model provides a sincere and comprehensive shift towards holistic event evaluation and should inspire future evaluation research that closely reflects Elkington’s (1997) ‘Triple Bottom Line’ accountability.

Although the traditional EIA may prove adequate for evaluation of three of the four principal policy objectives identified by Abelson (2011), Weisbrod and Weisbrod (1997) explain the quid pro quo involved in selecting the right kind of techniques available for assessing economic impacts, and how best to match the appropriate methods for different contexts. Their peer reviewed paper for the Economic Development Group highlights ‘seven deadly sins’ regarding EIA. Frechtling (1994) states that criteria for selection of economic models for EIA need to address relevance, coverage, efficiency, accuracy and transferability. This statement is echoed by Delpy and Ming (1997, p. 249) who write “…[when] evaluating an economic study, one needs to consider the study’s relevance, scope, efficiency, accuracy and verification”. Without careful consideration and objectification of these crucial factors and their inevitable trade-offs, the scope for misapplications of EIA are staggering. Crompton (1995) names eleven common misapplications of EIA and both his data, along with Weisbrod and Weisbrod’s (1997) ‘seven deadly sins’, are shown respectively in tables 1 and 2 below as they serve to succinctly summarise the collective, thematic and key grievances with EIA identified throughout the literature.  

Table 1 – Eleven sources of misapplication of event EIA (Crompton, 1995)

1.    Using sales instead of household income multipliers       2.    Misrepresenting employment multipliers       3.    Using incremental instead of normal multiplier coefficients       4.    Failing to accurately define the impacted area 5.     Including local attendees 6.     Failing to exclude ‘time switchers’ and ‘casuals’ 7.     Using ‘fudged’ multiplier coefficients 8.     Claiming total instead of marginal economic benefits 9.     Confusing turnover and multiplier 10.  Omitting opportunity costs 11.  Measuring only benefits and omitting costs

 Table 2 – ‘Seven deadly sins of EIA’ (Weisbrod & Weisbrod, 1997, p.11)

1. Confusing the economic role (gross effect) of a facility or project from its net impact on the economy of an area;       2. Adding together different measures of the same economic change (e.g., changes in business        sales and personal income);       3. Confusing study areas (e.g., neighbourhood, citywide, state and national effects);       4. Confusing time periods (e.g., immediate and eventual effects on economic growth);       5. Assuming that a facility’s capacity and its actual level of activity are the same;       6. Applying multipliers in situations where they don’t apply; and       7. Ignoring market effects on wages and land/building costs, which can also affect the economic competitiveness of an area

Conclusion

This essay has examined the rationale for event policy and the use of EIA for event evaluation. It has identified many weaknesses and flaws of EIA cited throughout the event evaluation literature. It has perhaps shown why academic response in addressing these flaws and weaknesses manifests slowly due to the complexities and challenges of objectifying data and selecting appropriate models and multipliers. It has explored and discussed the concepts of models and multipliers and highlighted the potential for misapplications of EIA. The contention surrounding EIA for effective event evaluation underpins the literature and highlights that the setting is ripe for research which answers the call of prominent academics in the field, and uses methodologies which foster a more holistic triple bottom line accountability.

References

Abelson, P. (2011). Evaluating major events and avoiding the mercantilist fallacy. Economic Papers, 30(1), 48-59. Retrieved March 26, 2017, from http://onlinelibrary.wiley.com.libproxy.murdoch.edu.au/doi/10.1111/j.1759-3441.2011.00096.x/epdf

Andersson, T.D., Armbrecht, J. and Lundberg, E. (2016) Triple impact assessments of the 2013 European athletics indoor championship in Gothenburg. Scandinavian Journal of Hospitality and Tourism, 16(2), 158-179. Retrieved March 24, 2017, from http://dx.doi.org/10.1080/15022250.2015.1108863

Archer, B. H. (1976). The uses and abuses of Multipliers. In W.W. Swart and T. Var (Eds.) Planning for Tourism Development: Quantitative Approaches (pp. 115-132). New York, NY: Praeger.

Archer, B.H. (1984). Economic impact: Misleading multiplier. Annals of Tourism Research, 11(3), 517-518.

Armbrecht, J. and Andersson, T.D. (2016) Subjects and objects of event impact analysis. Scandinavian Journal of Hospitality and Tourism, 16(2), 111-114. Retrieved march 24, 2017, from http://dx.doi.org/10.1080/15022250.2016.1162417

Blake, A. (2005). The Economic Impact of the London 2012 Olympics. Nottingham, UK: Nottingham University Business School.  Retrieved April 1, 2017, from http://hdl.handle.net/10453/19780

Breen, H., Bull, A., & Walo, M. (2001). A comparison of survey methods to estimate visitor expenditure at a local event. Tourism Management, 22, 473–479. Retrieved March 29, 2017, from http://dx.doi.org/10.1016/S0261-5177(01)00005-X

Chalip, L. (2003). Tourism and the Olympic Games. In M. Moragas, C. de Kennet, and   N. Puig. (Eds.), The Legacy of the Olympic Games 1984-2000. Lausanne: IOC.

Chalip, L. (2006). ‘Towards social leverage of sports events’. Journal of Sport & Tourism, 11(2), 109-127.

Chalip, L. and Leyns, A. (2002). Local business leveraging of a sport event: Managing an event for economic benefit, Journal of Sport Management, 17(3), 314-234. Retrieved March 26, 2017, from http://naspspa.org/AcuCustom/Sitename/Documents/DocumentItem/451.pdf

Chang, S., Kim, H.K. and Petrovcikova, M. (2015). Uses and Abuses of Economic Impact Studies in Tourism. Event Management, 19, 421-428. Retrieved March 24, 2017, from http://dx.doi.org/10.3727/152599515X14386220875002

Crompton, J. L., and S. L. McKay (1994). Measuring the Economic Impact of Festivals and Events: Some Myths, Misapplications and Ethical Dilemmas. Festival Management and Event Tourism, 2 (1), 33–43.

Crompton, J.L. (1995). Economic Impact Analysis of Sports Facilities and Events: Eleven Sources of Misapplication. Journal of Sport Management, 9, 14-35. Retrieved March 26, 2017, from  http://dx.doi.org/10.1123/jsm.9.1.14

Crompton, J.L. (2006). Economic Impact Studies: instruments for political shenanigans? Journal of Travel Research, 45, 67-82.

Delpy, L. and Ming, L. (1998). The art and science of conducting economic impact studies. Journal of Vacation Marketing 4(3), 230-254. Retrieved March 24, 2017, from http://dx.doi.org/10.1177/135676679800400303

Dixon, P. (2009). Comments on the productivity commission’s modelling of the economy: Wide effects of future automotive assistance. Economic Papers, 28(1), 11–18. Retrieved March 27, 2017, from http://dx.doi.org/10.1111/j.1759-3441.2009.00004.x

Dredge, D. and Jenkins, J. (2007). Tourism Planning and policy. Qld: John Wiley & Sons.          

Dwyer, L., Forsyth, P. and Spurr, R. (2004). Evaluating tourisms’ economic effects: new and old approaches. Tourism Management, 25(3), 307-317.

Dwyer, L., Forsyth, P. and Spurr, R. (2006). Assessing the Economic Impacts of Events: A Computable General Equilibrium Approach. Journal of Travel Research, 45, 59-66. Retrieved March 24, 2017, from http://journals.sagepub.com/doi/abs/10.1177/0047287506288907

Dwyer, L. and Jago, L. (2015). Economic evaluation of special events. In I. Yeoman, M. Robertson, U. McMahon-Beattie, E. Backer and K.A. Smith (Eds.), The Future of Events and Festivals (pp. 99-114).

Dwyer, L., Jago, L. and Forsyth, P. (2016). Economic evaluation of special events: Reconciling economic impact and cost–benefit analysis. Scandinavian Journal of Hospitality and Tourism, 16(2), 115-129. Retrieved March 24, 2017, from   http://dx.doi.org/10.1080/15022250.2015.1116404

Elkington, J. (1997). Cannibals with forks: The triple bottom line of 21st century business. Oxford, UK: Capstone.

Foley, M., McGillivray, D. and McPherson, G. (2012). Event policy: From theory to strategy. New York, NY: Routledge.

 Frechtling, D.C. (1994). Assessing the economic impacts of travel and tourism -Introduction to travel economic impact estimation. In J.R. B. Ritchie and C. R. Goeldner (Eds.), Travel, Tourism and Hospitality Research:  A Handbook for Managers and Researchers, (2^nd ed.) (ch.31). New York, NY: John Wiley & Sons. John Wiley & Sons.

Getz, D. and Frisby, W. (1988). Evaluating management effectiveness in community-run festivals. Journal of Travel research, 27(1), 22-27. Retrieved March 26, 2016 from http://journals.sagepub.com.libproxy.murdoch.edu.au/doi/abs/10.1177/004728758802700105

Getz, D. (1997). Event Management & Event Tourism. NSW: Cognizant Communication Corporation.

Getz, D. (2012). Event Studies: Theory, research and policy for planned events (2nd ed.). New York, NY: Routledge.

Harris, P. (1997). Limitations on the Use of Regional Economic Impact Multipliers by Practitioners: An application to the tourism industry. The Journal of Tourism Studies 8(2), 51-61. Retrieved March 24, 2017, from https://www.cabdirect.org/cabdirect/abstract/19981809937

Janeczko, B., Mules, T. and Ritchie, B. (2002). Estimating the economic impacts of festivals and events: a research guide. Cooperative Research Centre for Sustainable Tourism. Retrieved March 24, 2017, from  http://www.york.wa.gov.au/profiles/york/assets/clientdata/document-centre/2014_minutes/24_november_special/24_november_2014_-_9.4.1_estimate_econ_impacts_festivals_events_-_crc_sustainable_tourism_2014-11-23.pdf or

http://catalogue.nla.gov.au/Record/249176

Klijs, J., Heijman, W., Maris, D.K. and Bryon, J. (2012). Criteria for comparing economic models of tourism. Tourism Economics, 18(6), 1175-1202. Retrieved March 13, 2017 from http://dx.doi.org/10.5367/te.2012.0172

Kumar, J. and Hussain, K. (2014). A Review of Assessing the economic impact of Business tourism: issues And Approaches. International Journal of Hospitality & Tourism Systems, 7(2), 49-55. Retrieved March 24, 2017, from https://www.researchgate.net/publication/273287078_A_Review_of_Assessing_the_economic_impAct_of_Business_touRism_issues_And_AppRoAches

Masterman, G. (2009). Strategic Sports Event Management: Olympic Edition (2^nd ed.). Burlington, MA: Butterworth-Heinemann.

Pearce, G., Atkinson, D. and Mourato, S. (2006). Cost Benefit Analysis and the Environment: Recent developments. Paris: OECD Publishing.

Ritchie, J. B. (1984). Assessing the impact of hallmark events: Conceptual and research issues. Journal of Travel Research, 23(1), 2–11. Retrieved March 26, 2017, from http://dx.doi.org/10.1177/004728758402300101

Ritchie, J. B., & Beliveau, D. (1974). Hallmark events: An evaluation of a strategic response to seasonality in the travel market. Journal of Travel Research, 13(2), 14–20. Retrieved March 26, 2017, from http://dx.doi.org/10.1177/004728757401300202

Smith, A. (2014) Leveraging sport mega-events: new model or convenient justification? Journal of Policy Research in Tourism, Leisure and Events, 6(1), 15-30. Retrieved March 24, 2017, from http://dx.doi.org/10.1080/19407963.2013.823976

Testa, M.R. and Metter, M. (2017). Assessing Economic Impact as a Means for Event Efficacy: A proposed Model and Case Study. Event Management, 21, 61-70. Retrieved March 24, 2017, from https://doi.org/10.3727/152599517X14809630271113

Tindall, P. (2011). Economic Benefits of Rural Festivals and Questions of Geographical Scale: The Rusty Gromfest Surf Carnival. In C. Gibson and J. Connell, Festival Places: Revitalising Rural Australia (pp. 74-91). Bristol, UK: Channel View Publications.

Turco, D.M., & Kelsey, C.W. (1992). Conducting economic impact studies of recreation and parks special events. Arlington, VA: National Recreation & Park Association.

Wanhill, S. (1988). Tourism Multipliers under Capacity Constraints. Service Industries Journal, 8 (2),136–142. Retrieved March 31, 2017, from http://libproxy.murdoch.edu.au/login?url=http://search.proquest.com.libproxy.murdoch.edu.au/docview/203336250?accountid=12629

Weisbrod, G. and Weisbrod, B. (1997). Measuring Economic Impacts of Projects and Programs. Boston, MA: Economic Development Research Group. Retrieved March 24, 2017, from www.edrgroup.com/pdf/econ-impact-primer.pdf