Content uploaded by Riccardo Buccolieri
Author content
All content in this area was uploaded by Riccardo Buccolieri on Jun 11, 2014
Content may be subject to copyright.
Int. J. Environment and Pollution, Vol. 44, Nos. 1/2/3/4, 2011 403
Copyright © 2011 Inderscience Enterprises Ltd.
COST 732 in practice: the MUST model evaluation
exercise
Silvana Di Sabatino*
Dipartimento di Scienza dei Materiali,
University of Salento,
Via Monteroni, 73100 Lecce, Italy
E-mail: silvana.disabatino@unisalento.it
*Corresponding author
Riccardo Buccolieri
Dipartimento di Informatica,
Università “Ca’ Foscari” di Venezia,
Via Torino 155, 30172 Mestre, Italy
E-mail: riccardo.buccolieri@unive.it
and
Dipartimento di Scienza dei Materiali,
University of Salento,
Via Monteroni, 73100 Lecce, Italy
E-mail: riccardo.buccolieri@unisalento.it
Helge R. Olesen, Matthias Ketzel
and Ruwim Berkowicz
National Environmental Research Institute,
Aarhus University, P.O. Box 358,
Frederiksborgvej 399,
DK-4000 Roskilde, Denmark
E-mail: hro@dmu.dk
E-mail: ruwim@broendbybredbaand.dk
E-mail: mke@dmu.dk
E-mail: rb@dmu.dk
Jörg Franke
Faculty of Mechanical Engineering,
Department of Fluid- and Thermodynamics,
University of Siegen, Paul-Bonatz-Str. 9-11,
57076 Siegen, Germany
Fax: +49(0)2717402666
E-mail: joerg.franke@uni-siegen.de
404 S. Di Sabatino et al.
Michael Schatzmann,
K. Heinke Schlünzen and Bernd Leitl
Meteorological Institute (ZMK/ZMAW),
University of Hamburg,
Bundesstrasse 55,
D-20146 Hamburg, Germany
E-mail: schatzmann@zmaw.de
E-mail: heinke.schluenzen@zmaw.de
E-mail: bernd.leitl@zmaw.de
Rex Britter
Visiting Scientist, DUSP,
MIT. Boston, MA 02139, USA
E-mail: rb11@eng.cam.ac.uk
Carlos Borrego
and Ana Margarida Costa
CESAM & Department of Environment and Planning,
University of Aveiro,
3810-193 Aveiro, Portugal
E-mail: cborrego@ua.pt
E-mail: amcosta@ua.pt
Silvia Trini Castelli
Institute of Atmospheric Sciences and Climate (ISAC),
National Research Council (C.N.R.),
Corso Fiume 4, Torino – 10133, Italy
E-mail: S.TriniCastelli@isac.cnr.it
Tamir G. Reisin
Soreq Nuclear Research Center,
81800 Yavne, Israel
E-mail: treisin@soreq.gov.il
Antti Hellsten
Finnish Meteorological Institute,
Air Quality Research,
P.O. Box 503, FI-00101 Helsinki, Finland
Fax: +358919295403
E-mail: antti.hellsten@fmi.fi
COST 732 in practice: the MUST model evaluation exercise 405
Jarkko Saloranta
KONE Corporation,
P.O. Box 677, FI-05801 Hyvinkää, Finland
E-mail: jarkko.saloranta@kone.com
Nicolas Moussiopoulos
and Fotios Barmpas
Laboratory of Heat Transfer
and Environmental Engineering,
Dept. of Mechanical Engineering,
Aristotle University,
Box 483, GR-54124 Thessaloniki, Greece
E-mail: moussio@eng.auth.gr
E-mail: fotisb@aix.meng.auth.gr
Krzysztof Brzozowski
Department of Applied Computer Science,
University of Bielsko-Biala,
Willowa 2, 43309 Bielsko-Biala, Poland
E-mail: kbrzozowski@ath.eu
István Goricsán and Márton Balczò
Department of Fluid Mechanics,
Budapest University of Technology and Economics (BME),
1111, Bertalan L. 4-6,
Budapest, Hungary
E-mail: goricsan@ara.bme.hu
E-mail: balczo@ara.bme.hu
John G. Bartzis and George Efthimiou
Environmental Technology Laboratory,
Department of Mechanical Engineering,
University of Western Macedonia,
Sialvera & Bakola Str.,
50100, Kozani, Greece
E-mail: bartzis@uowm.gr
E-mail: gefthimiou@uowm.gr
406 S. Di Sabatino et al.
Jose Luis Santiago
and Alberto Martilli
Environment Department,
Research Center for Energy,
Environment and Technology (CIEMAT),
Avenida Complutense 22, 28040 Madrid, Spain
E-mail: jl.santiago@ciemat.es
E-mail: alberto.martilli@ciemat.es
Martin Piringer, Kathrin Baumann-Stanzer
and Marcus Hirtl
Central Institute for Meteorology and Geodynamics,
Division Customer Service,
Section Environmental Meteorology,
Hohe Warte 38, A-1190 Vienna, Austria
E-mail: martin.piringer@zamg.ac.at
E-mail: kathrin.baumann@zamg.ac.at
E-mail: marcus.hirtl@zamg.ac.at
Alexander A. Baklanov
Meteorological Research Division,
Danish Meteorological Institute,
Lyngbyvej 100,
DK-2100 Copenhagen Ø, Denmark
E-mail: alb@dmi.dk
Roman B. Nuterman
Meteorological Research Division,
Danish Meteorological Institute,
Lyngbyvej 100, DK-2100 Copenhagen Ø, Denmark
E-mail: ron@dmi.dk
and
Mechanics and Mathematics Faculty,
Department of Numerical Mathematics
and Computer Modelling,
Tomsk State University,
634050, 36 Lenin Ave., Tomsk, Russia
E-mail: nutrik@math.tsu.ru
COST 732 in practice: the MUST model evaluation exercise 407
Alexander V. Starchenko
Mechanics and Mathematics Faculty,
Department of Numerical Mathematics and Computer Modelling,
Tomsk State University,
634050, 36 Lenin Ave., Tomsk, Russia
E-mail: starch@math.tsu.ru
Abstract: The aim of this paper is to describe the use of a general methodology
tailored to the evaluation of micro-scale meteorological models applied to flow
and dispersion simulations in urban areas. This methodology, developed within
COST 732, has been tested through a large modelling exercise involving many
groups across Europe. The major test case used is the Mock Urban Setting Test
(MUST) experiment representing an idealised urban area. It is emphasised that
a full model evaluation is problem-dependent and requires several activities
including a statistical validation that requires a careful choice of the metrics for
the comparison with measurements.
Keywords: COST Action 732; CFD and non-CFD models; model evaluation;
MUST; idealised urban area.
Reference to this paper should be made as follows: Di Sabatino, S.,
Buccolieri, R., Olesen, H.R., Ketzel, M., Berkowicz, R., Franke, J.,
Schatzmann, M., Schlünzen, K.H., Leitl, B., Britter, R., Borrego, C.,
Costa, A.M., Trini Castelli, S., Reisin, T.G., Hellsten, A., Saloranta, J.,
Moussiopoulos, N., Barmpas, F., Brzozowski, K., Goricsán, I., Balczò, M.,
Bartzis, J., Efthimiou, G., Santiago, J.L., Martilli, A., Piringer, M.,
Baumann-Stanzer, K., Hirtl, M., Baklanov, A., Nuterman, R.B. and
Starchenko, A.V. (2011) ‘COST 732 in practice: the MUST model evaluation
exercise’, Int. J. Environment and Pollution, Vol. 44, Nos. 1/2/3/4, pp.403–418.
Biographical notes: Silvana Di Sabatino received her PhD (Engineering) from
the University of Cambridge (UK). She holds a MPhil Degree from the same
University and a MSc Degree in Physics from Bologna University. Currently,
she is Assistant Professor and Lecturer in Atmospheric Physics at the
Dipartimento di Scienza dei Materiali, University of Salento. Her research
interests are in the field of atmospheric dispersion over complex geometry,
urban boundary layer modelling and data analysis. She has published several
research papers in international journals and conference proceedings; she has to
her credit book chapters.
Riccardo Buccolieri received his PhD (Environmental Geophysics) from the
University of Messina (Italy) (Consortium of Universities of Messina, Palermo
and Lecce) working on flow and pollutant dispersion modelling in urban areas
by means of CFD and integral models. He is currently working as a Post-doc
Researcher at the Dipartimento di Informatica, Università di Venezia.
His research deals with the study of the effects of ambient wind direction,
building geometry and tree planting on flow and pollutant dispersion in the
urban environment. He is author of research studies in international journals,
conference proceedings as well as book chapters.
Helge Rørdam Olesen is a Senior Advisor at the Department of Atmospheric
Environment of the Danish National Environmental Research Institute, Aarhus
University. He has been working with development and implementation
of atmospheric short-range models, in particular the Danish OML model.
His main interest is model evaluation. An important aspect of his work has
408 S. Di Sabatino et al.
been efforts on an international level to establish common tools within
dispersion modelling. He is chairman of the European initiative on
Harmonisation within Atmospheric Dispersion Modelling for Regulatory
Purposes, which is responsible for the series of international harmonisation
conferences – resulting in the present issue of IJEP.
Matthias Ketzel is a Senior Scientist at the Department of Atmospheric
Environment of the Danish National Environmental Research Institute (NERI),
Aarhus University, Denmark. He has been working since 1993 with
atmospheric dispersion and aerosol dynamics modelling. He has more than
10 years of experience in working with the microscale model MISKAM and the
operational street canyon model OSPM. He received his PhD in 2004 from
Lund University, Sweden with a thesis on “Dispersion and Transformation of
Traffic Exhaust Particles in the Urban Atmosphere”. Personal web page:
http://www2.dmu.dk/atmosphericenvironment/Staff/ketzel.htm
Ruwim Berkowicz is retired senior scientist at the National Environmental
Research Institute in Denmark. His main research and work area has been
development of computerised air pollution dispersion models with special
emphasis on local scale modelling. He has been instrumental in development
of the Danish regulatory air quality model OML and the street canyon model
OSPM.
Jörg Franke received his PhD in Mechanical Engineering from the
University of Karlsruhe. He is a Senior Lecturer in the Department of
Fluid- and Thermodynamics, University of Siegen. His research interests
include the application of computational fluid dynamics in the fields of
building aerodynamics and pollution dispersion with special emphasis on
quality assurance of simulation results. He is the author of several research
studies in national and international journals, conference proceedings as well as
a book chapter.
Michael Schatzmann is Professor at the Meteorological Institute of the
University of Hamburg. He is in charge of the Technical Meteorology Division
which develops meso- and micro-scale meteorological models and operates a
boundary layer wind tunnel laboratory. He chairs action COST 732.
K. Heinke Schlünzen is Professor at the University of Hamburg and head of the
mesoscale and microscale modelling group. Her research interests are in the
field of meso- and micro-scale meteorological processes and phenomena
with particular emphasis on model evaluation and model uncertainty.
She investigates scale interaction, meso- and micro-scale dynamics
and chemistry, and the interaction of atmosphere, ocean, biosphere and
chemisphere. She has published more than 50 peer-reviewed papers
and numerous technical reports and conference papers.
Bernd Leitl is Professor and the Head of the Environmental Wind Tunnel
Laboratory (EWTL) at Hamburg University. He is working in the field
of environmental fluid mechanics for more than 20 years, mainly focusing on
physical modelling of atmospheric flow and dispersion phenomena in boundary
layer wind tunnels. For his work on LASER-optical, image-based concentration
measurements in large wind subsonic wind tunnels he was awarded a
Feodor-Lynen-Scholarship of the Alexander von Humboldt Foundation. He has
published more than 40 scientific papers and papers and authored or
co-authored numerous technical reports and conference papers.
Rex Britter is Visiting Scientist at the Department of Urban Studies and
Planning, MIT, Boston. He has been Professor of Environmental Fluid
COST 732 in practice: the MUST model evaluation exercise 409
Dynamics in the Department of Engineering at the University of Cambridge,
UK. His research interests include fundamental studies into turbulent fluid
dynamics, particularly those involving buoyancy. This is paralleled with
operational interests in the flow and dispersion of hazardous materials,
conventional pollutant dispersion problems in complex geometries such as
cities, formalised model evaluation procedures, urban air quality, sustainable
energy use in cities and security issues.
Carlos Borrego is Full Professor on Environmental Engineering, with focus on
Air Pollution. He has the Habilitation on Applied Environmental Sciences from
the University of Aveiro, PhD and MSc from the Free University of Brussels
and a Degree in Mechanical Engineering from the Technical University of
Lisbon. He has been involved since 1994 in 24 European projects
and integrates a number of international scientific boards and committees.
His research work includes nearly 400 scientific publications. He was the
Portuguese Minister of Environment and President of the European Council of
Ministers of Environment for the 1992 Rio Conference.
Ana Margarida Costa has a PhD in Sciences Applied to the Environment at the
University of Aveiro. She is a Researcher at the R&D Unit CESAM (Centre for
Environmental and Marine Studies) of the University of Aveiro, and carries
work in the field of traffic air pollutants dispersion in urban areas through the
development and application of numerical modelling tools. Her current
research interests also include the assessment of the outdoor air pollutants
transport to indoors and human exposure to air pollution. She is the author of
more than 65 scientific publications, including nine peer-reviewed papers in
scientific journals, 20 papers in International Symposia with referee, and the
participation in five book chapters.
Silvia Trini Castelli graduated in Physics in 1993 and received her PhD in
Geophysics in 1997. She is a Researcher at the Institute of Atmospheric
Sciences and Climate of the C.N.R. (Torino, Italy) and she collaborates as
short-term Professor at the University of Piemonte Orientale. Her main
research fields are atmospheric physics, planetary boundary layer meteorology,
turbulence, atmospheric dynamics and dispersion modelling. She is involved
in several national and international research projects, is author of more than
50 papers published on international journals and peer-reviewed conference
proceedings and she has presented more than 60 papers at international and
national meetings.
Tamir G. Reisin received his BSc and MSc Degrees in Physics from the
Technion – Israel Institute for Technology and a PhD in Atmospheric Physics
in 1995 from Tel Aviv University. After a one year residence as Visiting
Scholar at the Geophysical Fluid Dynamics Laboratory at Princeton University
he returned to Tel Aviv University as Researcher where he stayed until 1999.
Since then he has been a Researcher at the Applied Physics Division at the
Soreq Nuclear Research Centre. His main research fields are cloud physics,
microphysical processes numerical modelling, and pollutants dispersion.
He published more than 70 papers in scientific journals and conference
proceedings.
Antti Hellsten received his DSc (Tech.) Degree in Aeronautical Engineering
at Helsinki University of Technology. He is currently a Senior Research
Scientist at Finnish Meteorological Institute. His research interests are
related to simulation and modelling of turbulent flows in various technical and
environmental applications. He has published many research papers in
international journals and conference proceedings.
410 S. Di Sabatino et al.
Jarkko Saloranta received his MSc (Tech.) Degree in Aeronautical Engineering
at Helsinki University of Technology in 2006. Up to autumn 2008 he worked as
a research scientist at Helsinki University of Technology. He is currently
working as a Design Engineer at KONE Corporation R&D unit.
Nicolas Moussiopoulos is a Full Professor at Aristotle University’s Mechanical
Engineering Department and the Director of the Laboratory of Heat Transfer
and Environmental Engineering. At present he is also the Dean of the
University’s School of Engineering. Since 1996, he is a Honorary Professor at
Karlsruhe University’s School of Mechanical Engineering. His research work
deals primarily with air pollution abatement and waste management. He is a
Member of the German Academy of Natural Scientists Leopoldina and in 2002
he was awarded the Order of Merit of the Federal Republic of Germany.
Since 2009, he represents Greece in the European Commission’s Programme
Committee “Regions of Knowledge, Research Potential and Coherent
Development of Policies”.
Fotios Barmpas graduated from the University of Manchester as an Aerospace
Engineer in 1997. He received an MSc in “Applied Mathematics and Fluid
Mechanics” also from the University of Manchester in 1999. Since then he has
held a research engineering position at the Laboratory of Heat Transfer and
Environmental Engineering, AUTh, in the field of Computational Fluid
Dynamics (CFD) and Heat Transfer. He has worked on the optimisation of heat
exchangers in future commercial airliners aero-engines through CFD analysis
as well as on wind flow and dispersion of pollutants in built-up areas for
various European projects through the application of both numerical and wind
tunnel modelling techniques. He has been involved in several large EC projects
and has contributed to several major environmental impact assessment studies.
Fotios Barmpas has been involved in several scientific publications, including
several articles in peer reviewed journals.
Krzysztof Brzozowski received his PhD in Mechanical Engineering from
University of Bielsko-Biala, Poland. He is a Lecturer at the Department
of Applied Computer Science, University of Bielsko-Biala. His main research
area is concerned with applications of computational methods to modelling
of car exhaust pollutant emission and dispersion. He is the author of books and
several papers in national and international journals.
István Goricsán received his PhD in Mechanical Engineering at the Budapest
University of Technology and Economics (BME). His research interests
include issues related to dispersion in atmospheric boundary layers, wind
tunnel measurement of wind loads acting on buildings and structures, as well as
pollutant transport related CFD.
Márton Balczó graduated as Mechanical Engineer 2001 from the BME and the
University of Karlsruhe and is research assistant at the Department of Fluid
Mechanics of the Budapest University of Technology and Economics (BME).
His research area is pollutant transport in urban environment.
John Bartzis received his Diploma in Mechanical and Electrical Engineering
from the National Technical University of Athens, Greece (1970), his MS in
Nuclear Engineering from the MIT, USA (1975), and his PhD in Nuclear
Engineering from the MIT, USA (1977). He is currently a Professor, and
Director of the Energy and Environment Sector and Director of the
Environmental Technology Laboratory UOWM. He is also Elected
Vice-President of COST Senior Officials Committee and Member of the
Safety Committee of Greek Research Nuclear Reactor. He was also Elected
Vice-Chairman of COST Senior Officials Committee, Chairman of the
COST 732 in practice: the MUST model evaluation exercise 411
COST Strategy Issues Group, Member of JAF Committee of COST,
National Representative of COST Senior Official Committee and National
representative in several EU and OECD Committees in the field of
Nuclear Energy.
George Efthimiou received his degree in Mechanical Engineering
at the UOWM, Greece. He is a PhD student at the Department of Mechanical
Engineering, UOWM, Greece. His research interests include the microscale
modelling of environmental flows and pollutant dispersion with special
emphasis on individual exposure in short term release episodes.
Jose Luis Santiago holds a MSc Degree in Physics from University of Cordoba
(UCO) and a PhD from the Technical University of Madrid (UPM).
He obtained the Extraordinary Doctoral Prize from the UPM in 2005/2006.
Currently, he is a Researcher at the Environment Department of CIEMAT
(Spain). His research subjects are urban meteorology and atmospheric
dispersion modelling in street canyons. He have published several research
papers at international journals, conference proceeding, technical reports and
book chapters.
Alberto Martilli graduated in Physics in 1995 (Università Statale of Milano,
Italy) with a dissertation on numerical modelling of mesoscale circulations over
complex terrains. In 1996, he moved to the Swiss Federal Institute
of Technology (EPFL, Lausanne), where, in 2001, obtained his PhD with a
dissertation on the development of a new urban surface parameterisation
for mesoscale models. From 2001 to 2004 he worked as Post-doctoral fellow
at the University of British Columbia (Vancouver, Canada), on mesoscale air
pollution modelling. Since 2005 he is researcher at CIEMAT (Spain), where he
works on micro and mesoscale atmospheric modelling over urban areas.
Martin Piringer has a Doctorate in Meteorology from the University of Vienna,
with a thesis (in 1980) on a concept and first results of a fine-mesh boundary
layer model in the Alps. He is today head of the Section Environmental
Meteorology in the Division Customer Service at the Central Institute for
Meteorology and Geodynamics and has published 24 refereed papers (focusing
on urban boundary layers and odor pollution) as well as chapters of three
books. His research interests are mainly vertical profile measurements in the
boundary layer (both in complex terrain and the urban environment) as well as
odor dispersion.
Kathrin Baumann-Stanzer has a Doctorate in Meteorology from the University
of Vienna, with a thesis on the quality assurance of wind profiler data.
She has published in national and international journals, 21 refereed papers on
various aspects of applied boundary layer meteorology and air pollution
modelling. She is senior scientist at the Central Institute for Meteorology and
Geodynamics and leader of the emergency response group in the Section
Environmental Meteorology. Her research interests are related to air pollution
modelling on the local and regional scale – for assessment studies as well as
hazardous release scenarios.
Marcus Hirtl holds a Master Degree in Meteorology. He is a scientist in the
Section Environmental Meteorology at the Central Institute for Meteorology
and Geodynamics. His main interests are ozone forecasting and air pollution
modelling at all scales. He has published 3 refereed papers and is actively
participating at international conferences and working groups.
Alexander A. Baklanov, Prof. (2008), Dr. Sci. (1998), PhD (1983), MS (1979),
is senior scientist at Research Department of Danish Meteorological Institute
412 S. Di Sabatino et al.
and deputy director of Danish strategic research Centre for Energy,
Environment and Health (CEEH). His research interests are air pollution and
aerosol dynamics modelling, boundary layer meteorology, urban meteorology
and climatology, environmental impact and risk assessments. He has more than
300 scientific publications, including 11 books and about 150 journal papers.
Roman B. Nuterman received his PhD in Fluid Mechanics at the Tomsk State
University, Russia. He is PostDoc at the Research Department of Danish
Meteorological Institute (Center for Energy, Environment and Health, CEEH).
His research interests are computational fluid dynamics, air pollution and
boundary layer meteorology. He is the author of several research studies in
national and international journals and conference proceedings.
Alexander V. Starchenko received his PhD at Tomsk State University (TSU),
Russia. After that he worked at Scientific Research Institute of Applied
Mathematics and Mechanics at Tomsk State University. He received his
Doctoral Degree in Physics and Mathematics from TSU. Presently he is a
Professor of Mechanics and Mathematics Faculty and Head of the Numerical
Mathematics and Computer Modelling Department of TSU. His research
interests are fluid mechanics, numerical methods, boundary layer meteorology,
urban meteorology and climatology. He has published one monograph and
about 100 journal papers.
1 Introduction
COST Action 732 (2005–2009) was launched with the intent of improving and assuring
the quality (fitness-for-purpose) of micro-scale meteorological models that are used for
predicting flow and dispersion in urban or industrial environments. COST 732 started in
July 2005 with a joint ESF/COST 732 Exploratory Workshop on Quality Assurance of
Micro-Scale Meteorological Models held in Hamburg. The eventual impact of COST 732
is dependent upon whether the evaluation procedures suggested by the Action are widely
accepted by the scientific community of model developers and model users. In May 2007
the first version of the evaluation procedure was released in order to provide a basis for
discussions within the community. One of the main aims of COST 732 is that of
establishing best practice methodologies and a standardisation of CFD modelling practise
when used for meteorological applications within urban areas. The evaluation procedure
proposed by COST 732 is presented in three documents that are publicly available on line
at http://www.mi.uni-hamburg.de/Official-Documents.5849.0.html
• Background and Justification Document to support the Model Evaluation Guidance
and Protocol, Version 1, May 2007 (Britter and Schatzmann, 2007a).
• Model Evaluation Guidance and Protocol Document, Version 1, May 2007 – a
stand-alone document to assist the setting up and executing of a model evaluation
exercise (Britter and Schatzmann, 2007b).
• Best Practice Guideline for the CFD simulation of flows in the urban environment,
Version 1, May 2007 – based on published guidelines and recommendations, which
mainly deal with prediction of the statistically steady mean flow and turbulence in
urban areas under conditions of neutral density stratification (Franke et al., 2007).
COST 732 in practice: the MUST model evaluation exercise 413
The recommendations given in the documents listed above have been tested by COST
732 itself. The Mock Urban Setting Test (MUST), (Biltoft, 2001; Yee and Biltoft, 2004)
which comprises field and wind tunnel experiments from flow and dispersion
experiments carried out within and above a simulated urban setting made up from
120 standard size shipping containers, was selected for initial studies. The wind tunnel
measurements within a scaled model (1 : 75) of that configuration were carried out at the
University of Hamburg. (Bezpalcova, 2007). So far, several groups of numerical
modellers (with most using CFD models) have simulated the wind tunnel MUST
experiments following the model evaluation guideline. The experiments used were those
with two main wind directions, 0° and –45° (and these correspond to 270° and 315°,
respectively, in meteorological terminology) of the approaching flow. This study was
launched in Athens in October 2006 and is the basis for COST 732 testing the evaluation
procedures for urban flow and transport models. Attention is given to determining the
quality (fitness-for-purpose) of the model.
2 Methodology
Several CFD models have been used by different groups from many European countries.
They are: Miskam, Fluent, ADREA, Star-CD, Finflo, Cfx, Mitras, Tsu/M2UE, VADIS,
Code_Saturne. Also non-CFD models, such as Lasat, ADMS-Urban, RAMS, OML,
ESCAPE, CALPUFF, have received attention within COST 732. For comparison of
numerical results with experimental data, both qualitative and quantitative approaches
have been chosen. There is a common understanding that exploratory qualitative data
analysis using graphical comparison between model and data and an intercomparison
among models gives a simple, useful and transparent way of showing the strengths and
weaknesses of models.
For the evaluation of a model both qualitative (through profiles and contours) and
quantitative (through statistical analysis) approaches are needed, otherwise statistical
parameters alone could obscure deficiencies of the model.
In our proposed methodology model results needs to be analysed in a combined way
by means of
• contours of velocity components, Turbulent Kinetic Energy (TKE) and Reynolds
stress components
• vertical and horizontal profiles of velocity components and TKE
• profiles of dimensionless concentration. In the example provided we only use
the –45° approach flow case as concentration measurements were not available
for the 0° case
• statistical analyses.
The first three are essentially a qualitative analysis while the fourth is quantitative. In our
methodology, model results are quantitatively evaluated using direct point-by-point
comparisons with wind tunnel data focusing on the mean velocity components, on the
TKE, on the Reynolds stresses and on the pollutant concentrations. This approach was
preferred over the alternative of using manipulated data such as estimating a maximum
concentration on an arc and using it for model comparison purposes. To assess model
414 S. Di Sabatino et al.
performance several statistical measures can be used, such as the Fractional Bias (FB),
the Normalised Mean Square Error (NMSE), the fraction of predictions within a factor of
two of observations (FAC2), the Geometric Mean (MG) bias and the geometric variance
(VG). Typical magnitudes of the above performance measures and estimates of model
acceptance criteria have been summarised by Chang and Hanna (2004) based on
extensive experience with evaluating many models with many field data sets.
The commonly accepted values for ‘state of the art’ model performance
are: –0.3 < FB < 0.3; NMSE < 4; FAC2 ≥ 0.5; MG < 1; VG < 1.5. Also the hit rate
evaluation test (VDI, 2005) should be performed using a fractional deviation D = 0.25
and specific absolute deviations W for the different variables analysed (the hit rate must
not fall below 66% for the comparison with wind tunnel data).
In this perspective, assuming that the set of models involved in COST 732 is a
representative sample of the micro-scale models currently available and widely used,
at least in a European context, with the MUST exercise the Action’s intention is to
suggest criteria for the ‘state of the art’ based on the model results. The state of the art is
a dynamical concept; models constantly improve and the state of the art consequently
changes. So the methodology which the Action is following will contain a procedure to
update the criteria, so that if, in the future, new models are run using the COST
732-MUST case or other data, the value of the metrics reflecting the state of the art will
be modified. Currently, such criteria are valid only for the COST 732-MUST dataset but
future model comparisons with other datasets under COST 732 will provide more
comprehensive model evaluation.
A somewhat different question concerns the ‘fitness for purpose’ criteria as these
change with the intended purpose of the model. An important point to be addressed by a
model user is whether the ‘state of the art’ will satisfy the ‘fitness for purpose’ criteria for
the particular purpose of the modeller.
3 Results
Vertical and horizontal profiles of wind tunnel data and the results from the various
model simulations have been collected in Excel spreadsheets that include a macro tool,
which allows easy graphical comparisons. The tool was developed within this Action by
Berkowicz et al. (2007). This tool was found to be extremely useful for exploratory data
analysis to highlight both large errors and subtle differences among the models.
In this section we present, as an example of the qualitative evaluation, some profiles
of the horizontal velocity component along the x-axis (U) and the vertical velocity
component (W) for the 0° case (normalised with the reference wind velocity of
undisturbed flow Uref ). We also show concentration profiles for the –45° approach flow
case. Concentration values are expressed in a non-dimensional form as follows:
2
ref
*CU H
CQ
= (4)
where C is the calculated concentration, H the building height and Q the emission rate.
COST 732 in practice: the MUST model evaluation exercise 415
3.1 Wind profiles
For the 0° approach flow case, Figure 1 shows example vertical profiles at one
measurement point in the spanwise street canyon at the middle of the array.
From Figure 1, we note that the qualitative behaviour of the models is different. Some of
them seem to underestimate U/Uref in the layer occupied by the buildings, while others
overestimate the velocity. Table 1 shows the validation metrics from two of the models
involved. These metrics have been calculated considering all vertical profiles available.
The agreement between models and experiments is good for the U velocity component
with all metrics within the range or very close to the acceptance criteria limits. It should
be noted that FB and NMSE parameters are without meaning for velocity components or
variables which may assume both positive and negative values. Overall all models are not
able to predict well the W component and they tend to underestimate it.
Figure 1 0° approach flow case: (a) sketch of the array showing the position (vertical line) of
measurement profiles in the street canyon and (b) examples of modelled and measured
vertical profiles of U/Uref and W/Uref (where mod indicates modelled results and WT
indicates wind tunnel data)
(a)
(b)
416 S. Di Sabatino et al.
Table1 0° approach flow case: statistical quantities of U/Uref and W/Uref obtained from some
models against wind tunnel data for all vertical profiles available
MODEL FAC2 (U/Uref; W/Uref) Hit rate (U/Uref; W/Uref)
B 0.88; 0.30 0.61; 0.24
C 0.92; 0.24 0.66; 0.17
3.2 Tracer concentration
The qualitative behaviour of the pollutant plume predicted by the models compares well
with wind tunnel data. As an example Figure 2 shows concentration results in the middle
of the array for the –45° approach flow case plotted at a constant height z = 0.5 H (where
H is the height of the building).
Table 2 shows, as an example, the validation metrics for concentration from two of
the models involved for all measurements points (256). The agreement is good for the
two models with all metrics within or very close to the acceptance limits.
The Action has also collected results from the statistical analysis for all the models
participating to the MUST exercise. Differences in the metrics for different models for
the MUST data and for other datasets and the criteria for the ‘state of the art’ and for
‘fitness for purpose’ for typical purposes have been discussed. This has allowed to
formulate a Best Practice Guideline for using the MUST case and to revise the Model
Evaluation Guidance and Protocol Documents. Further detailed qualitative and
quantitative results (Schatzmann et al., 2009) have been presented during several
conferences and will be presented over the next years.
Figure 2 –45° approach flow case: (a) top view of the array, position of the source (black square
in the upper part) and measurements points (left part). Wind blows from the top and
(b) examples of modelled and measured C* values along an horizontal profile
perpendicular to the wind direction (black rhombi in Figure 2(a))
(a) (b)
COST 732 in practice: the MUST model evaluation exercise 417
Table 2 –45° approach flow case: statistical quantities of dimensionless concentrations C*
obtained from some models against wind tunnel data for all measurements points
MODEL FB FAC2 Hit rate
B –0.26 0.97 0.68
C –0.39 0.94 0.64
4 Lessons learnt and conclusions
The first qualitative comparison carried out in the COST 732 shows that flow and
concentration model results compare relatively well with the measurements.
The prediction for the U velocity component is better than for the W component.
The Excel tool developed within the COST 732 has allowed us to make detailed studies
of the differences in model results where the differences are not obscured by differences
in presentation. Correct specification of the inlet profiles and specific aspects of
two-equation turbulence models were thought to require further investigation.
The Action has finalised the MUST exercise and has suggested the best approach
for further model evaluation for the standardisation of CFD modelling practise for
micro-scale meteorological applications. This includes a critical review and refinement of
the numerical results. This has been done by using outcome from some working groups
which were formed during the Action to investigate specific aspects including boundary
conditions, statistical measures and non-CFD model evaluations.
Acknowledgements
Thanks are due to DTRA/DPG for providing MUST data for use in COST732.
References
Berkowicz, R., Olesen, H. and Ketzel, M. (2007) http://www.dmu.dk/en/air/models/
background/must/
Bezpalcova, K. (2007) Physical Modelling of Flow and Diffusion in Urban Canopy, PhD Thesis,
Faculty of Mathematics and Physics, Charles University in Prague.
Biltoft, C.A. (2001) Customer Report for Mock Urban Setting Test, DPG Document No. WDTCFR-
01-121, West Desert Test Center, US Army Dugway Proving Ground, Dugway, Utah, p.58.
Britter, R. and Schatzmann, M. (Eds.) (2007a) Background and Justification Document to Support
the Model Evaluation Guidance and Protocol Document, COST Office Brussels, Belgium.
Britter, R. and Schatzmann, M. (Eds.) (2007b) Model Evaluation Guidance and Protocol
Document, COST Office Brussels, Belgium.
Chang, J. and Hanna, S. (2004) ‘Air quality model performance evaluation’, Meteorol. Atmos.
Phys., Vol. 87, pp.167–196.
COST Action 732 (2005–2009) Quality Assurance and Improvement of Micro-Scale
Meteorological Models, http://www.mi.uni-hamburg.de/Home.484.0.html
Franke, J., Hellsten, A., Schlünzen, K.H. and Carissimo, B. (Eds.) (2007) Best Practice Guideline
for the CFD Simulation of Flows in the Urban Environment, COST Office Brussels, Belgium.
418 S. Di Sabatino et al.
Schatzmann, M., Olesen, H. and Franke, J. (Eds.) (2009) COST 732 Model Evaluation Case
Studies: Approach and Results, COST Office, Brussels, Belgium.
VDI (2005) VDI Guideline 3783 Part 9: 2005-11, Environmental Meteorology – Prognostic
Micro-scale Wind Field Models – Evaluation for Flow Around Buildings and Obstacles,
Beuth Verlag, Berlin.
Yee, E. and Biltoft, C. (2004) ‘Concentration fluctuation measurements in a plume dispersing
through a regular array of obstacles’, Boundary Layer Meteorology, Vol. 111, pp.363–415.
