Content uploaded by Ricardo Vinuesa
Author content
All content in this area was uploaded by Ricardo Vinuesa on Jan 14, 2022
Content may be subject to copyright.
Review
Flow control in wings and discovery of novel
approaches via deep reinforcement learning
R. Vinuesa1,∗, O. Lehmkuhl 2, A. Lozano-Durán3and J. Rabault4,5
1FLOW, Engineering Mechanics, KTH Royal Institute of Technology, Stockholm, Sweden.
2Barcelona Supercomputing Center, Barcelona, Spain.
3
Dept. of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
4IT Department, Norwegian Meteorological Institute, Oslo, Norway.
5Independent researcher.
*Email for correspondence: rvinuesa@mech.kth.se
Version January 4, 2022 submitted to Fluids
Abstract:
In this review we summarize existing trends of flow control used to improve the
1
aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting
2
with flat-plate geometries and building towards the more complicated flow around wings. Then, we
3
discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic
4
efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various
5
levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow
6
control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the
7
potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.8
Keywords:
turbulence; flow control; simulation; aerodynamics; machine learning; deep
9
reinforcement learning10
1. Introduction11
Over the past decades, aviation has become an essential component of today’s globalized world:
12
before the current pandemic of coronavirus disease 2019 (COVID-19), over 100,000 flights took off
13
everyday allowing the transportation of people and goods, and the establishment of global commercial
14
relations. Despite the significant impact of the pandemic on the aviation sector, a number of studies
15
indicate that after the pandemic its relevance in the transportation mix will be similar to that before
16
COVID-19 [
1
,
2
]. Aviation alone is responsible for 12% of the carbon dioxide emissions from the whole
17
transportation sector, and for 3% of the total CO
2
emissions in the world [
3
]. On the other hand, fuel
18
represents around 40% of the costs in regular airlines, corresponding to hundreds of millions of dollars
19
spent yearly [
4
]. This has important implications in the context of the Sustainable Development Goals
20
(SDGs) 9 (on sustainable infrastructure) and 11 (on sustainable cities), as well as an important impact on
21
SDGs 3 (on health) and 13 (on climate change) from the United Nations (UN) 2030 Agenda [
5
–
7
]. Due
22
to the major environmental and economical impacts associated with aviation, it is desirable to improve
23
the aerodynamic performance of airplane wings, with the aim of reducing the fuel consumption and
24
emissions associated with air travel.25
In order to develop more efficient wings it is necessary to reduce the losses associated with their
26
movement within the surrounding fluid. This implies reducing the force parallel to the incoming
27
flow (the drag), and one of the strategies to achieve such a reduction is to perform flow control. A
28
wide range of methods aimed at controlling the flow to reduce the drag have been reported and
29
some have documented net-energy savings, i.e. taking into account the energy spent on the control,
30
as documented by Fahland et al. [
8
]. These strategies include passive methods, such as riblets [
9
],
31
Submitted to Fluids, pages 1 – 16 www.mdpi.com/journal/fluids
Version January 4, 2022 submitted to Fluids 2 of 16
which are drag-reducing surfaces proven successful in passenger aircraft [
10
], and active techniques, in
32
which the drag-reduction effect is achieved through an action which requires additional energy to be
33
transferred to the flow [
11
]. When the control action is determined based on the instantaneous flow
34
state, sensing is required and the control strategy is denoted as reactive [
12
], whereas in predetermined
35
strategies the control is decided a-priori [
13
]. Predetermined strategies are appealing due to their
36
simple implementation; however, reactive methods are the ones which have the highest potential to
37
reduce drag. Note that currently-available reactive controls, such as opposition control [
14
,
15
] through
38
blowing and suction at the wall, are simply based on an ad-hoc gain to relate the sensed velocity
39
fluctuation and the actuation (see §1.1 for more details). It is important to highlight that the total drag
40
in subsonic transport aircraft can be decomposed essentially into four components, namely: friction
41
drag, lift-induced drag (where the lift is the aerodynamic force perpendicular to the incoming stream),
42
wave drag and parasite drag. Whereas the two latter only represent up to 10% of the total drag, the first
43
and second are responsible for around 50% and 40% respectively in cruise conditions [
16
,
17
]. Although
44
the continuous design optimization carried out over the past decades has led to reductions of the drag,
45
new innovative approaches are required in order to obtain significant improvements in aerodynamic
46
performance [18]. In the following, we will discuss in more detail different types of active control.47
1.1. Active control of turbulent flows48
A widely-used method of predetermined active flow control is uniform blowing/suction. The first
49
wind-tunnel experiments using the micro-blowing technique (MBT) [
19
] reported that it is possible to
50
achieve a significant drag reduction with relatively moderate blowing, as well as to have net-energy
51
savings in full-scale applications. More recent studies have confirmed this possibility, investigating
52
the effects of MBT on more complex geometries and on adverse-pressure-gradient (APG) turbulent
53
boundary layers (TBLs). A detailed description of the MBT technique is provided by Hwang [
20
], and
54
Kornilov [
21
] discusses more recent developments, particularly regarding experimental results. On the
55
other hand, high-fidelity numerical simulations have been used to better characterize the interaction
56
between control and wall-bounded turbulence. One of the first numerical studies investigating TBLs
57
with blowing and suction is that of Park and Choi [
22
], who employed direct numerical simulations
58
(DNSs, in which all turbulent scales are resolved) and considered a Reynolds number based on
59
displacement thickness
δ∗
and freestream velocity
U∞
of
Reδ∗=
500. Kametani and Fukagata [
23
]
60
performed DNS of a zero-pressure-gradient (ZPG) TBL with blowing and suction at Reynolds numbers
61
based on momentum thickness
Reθ
between 300 and 700, with intensities up to 1% of
U∞
. They
62
also analyzed the energy input associated with uniform blowing to estimate the upper bound of
63
control efficiency, and they confirmed that it is theoretically possible to achieve net-energy savings. As
64
expected, uniform suction has opposite effects. Later, other numerical simulations (more relevant to
65
the full-scale conditions) have been performed at higher Reynolds numbers. For instance, Kametani et
66
al. [
24
] carried out high-resolution large-eddy simulations (LESs, where only the smallest turbulent
67
scales are modelled) of a ZPG TBL at
Reθ=
2, 500, considering blowing and suction with an intensity
68
of 0.1% of the freestream velocity. These authors achieved more than 10% drag reduction despite the
69
relatively low blowing intensity.70
The numerical studies discussed above are focused on the description of the effect of blowing on a
71
spatially-developing ZPG TBL, which is an idealized study case. Firstly, more realistic scenarios exhibit
72
more complex turbulent flows [
25
], including as pressure-gradient TBLs [
26
] and finite aerodynamic
73
bodies [
27
,
28
]. In these cases it is not trivial to generalize the control techniques. Secondly, the
74
skin-friction reduction is beneficial in engineering applications only if it corresponds to a reduction of
75
total drag (which includes additional components as discussed above), and/or to an improvement of
76
the aerodynamic efficiency (defined as the lift-to-drag ratio
L/D
). For these reasons, the two following
77
experimental studies on the effects of blowing and suction in airfoils are of particular relevance. We
78
first discuss the work by Eto et al. [
29
], who applied a blowing intensity of 0.14% of
U∞
to the suction
79
side of a Clark-Y airfoil at
Rec=
1, 600, 000 (where
Rec
is the Reynolds number based on
U∞
and
80
Version January 4, 2022 submitted to Fluids 3 of 16
Figure 1.
Instantaneous visualization of coherent vortical structures [
34
] around a NACA4412 wing
section at
Rec=
400, 000, colored by streamwise velocity ranging from (dark blue)
−
0.2 to (dark red)
1.7. The yellow line indicates the extent of the control region (with uniform blowing on the suction
side), whereas the red one denotes the tripping location [
35
,
36
]. Figure extracted from Ref. [
32
], with
permission of the publisher (Springer Nature).
wing chord
c
). They observed a local reduction of the skin friction between 20 and 40%, but they also
81
reported an increase of the total drag. On the other hand, Kornilov et al. [
30
] carried out experiments on
82
a NACA0012 airfoil at
Rec=
700, 000, applying blowing and suction over both sides of the airfoil. They
83
confirmed that blowing over the suction side does not reduce the total drag, but they also observed
84
that blowing over the pressure side and suction over suction side have a beneficial effect, achieving a
85
reduction of total drag of around 10%. This highlights the additional difficulty of performing control
86
in wings, where the various contributions to the total drag are tightly coupled. One of the first
87
high-fidelity simulations of turbulent wings with control was conduced by Vinuesa and Schlatter [
31
]
88
in 2017. In that work, as well as in more recent studies [
32
], high-resolution LES was used to study the
89
turbulent flow around a NACA4412 wing section, up to
Rec=
400, 000, where different combinations
90
of blowing and suction rates over the suction and pressure sides were applied. Using predetermined
91
active flow control, they achieved a maximum increase of the aerodynamic efficiency of 11% [
32
]. In
92
Figure 1we show the effect of applying uniform blowing on the suction side of the wing section,
93
which leads to an increase of boundary-layer thickness and turbulence activity far from the wall. This
94
produces a reduction of the wall-shear stress and an increase of the pressure drag, leading to a higher
95
total drag; note that the opposite holds for uniform suction [32]. Another interesting numerical work96
is that of Albers et al. [
33
], who performed LES on an airfoil at
Rec=
400, 000 to assess the effect of
97
transversal surface waves, reporting a drag reduction of 7.5%. As discussed below, it is possible to
98
obtain more sophisticated and efficient control strategies by sensing the flow and exploiting all the
99
available information on its state, i.e., by performing reactive flow control.100
A widely-used method of reactive control is the so-called opposition control, where suction and
blowing are introduced at the wall with the aim of suppressing the sweep and ejection events in the
near-wall region, so as to reduce the skin friction [
14
]. Essentially, the velocity imposed at the wall
vw
should be opposite to the wall-normal velocity
v
at a certain sensing plane
ys
, according to the
equation:
vw(x, 0, z,t) = −α[v(x,ys,z,t)−V(ys,t)]. (1)
Note that here
x
,
y
and
z
denote the streamwise, wall-normal and spanwise coordinates,
t
is the time,
101
α
is a positive constant and
V
is the instantaneous wall-normal velocity averaged over the control area.
102
Subtracting this term ensures a zero-net-mass-flux condition at the wall. It is important to note that,
103
through this equation, the control aims at opposing the fluctuations at a certain wall-normal location,
104
Version January 4, 2022 submitted to Fluids 4 of 16
which is typically around the near-wall fluctuation peak, i.e.
y+
s=
15. The superscript ‘+’ denotes
105
inner scaling, in terms of the viscous length
`∗=ν/uτ
, where
ν
is the fluid kinematic viscosity and
106
uτ=pτw/ρ
is the friction velocity (with
τw
being the wall-shear stress and
ρ
the fluid density). The
107
constant
α
is set empirically, which means that the resulting control law is relatively simple. Despite
108
this simplicity, Stroh et al. [
15
] reported drag-reduction rates of around 20% in turbulent channels
109
and boundary layers up to
Reτ'
660 (which is the friction Reynolds number, based on the 99%
110
boundary-layer thickness
δ99
and the friction velocity
uτ
). However, it may be possible to obtain more
111
sophisticated control laws by formulating an optimization problem, as discussed in §3.112
1.2. Active control of separation113
Several studies exploring the capabilities of active-flow-control (AFC) actuators in high-lift devices
114
with massive separation (i.e. suction and blowing, sweeping jets, fluidic oscillators, plasma actuators,
115
synthetic jets) have been conducted in the literature. A brief review on the state of the art of active flow
116
control techniques for civil aircrafts can be found Batikh et al. [
37
]. Khodadoust and Washburn [
38
]
117
conducted wind-tunnel measurements on a high-lift device fitted with AFC actuators. They observed
118
that the application of small amount of suction and blowing increased the lift performance. Khün et
119
al. [
39
] simulated a 3D high-lift wing with constant blowing using RANS. The results showed that
120
blowing can be beneficial to suppress massive separation in the flap. Radespiel et al. [
40
] reviewed
121
different techniques for AFC using constant blowing showing that tangential blowing can be promising
122
at increasing the lift at high angles of attack. Fricke et al. [
41
] simulated the AFC by means of pulsed
123
blowing to control flow separations in the wing engine junction with RANS. Later, Schloesser et al.
124
[
42
] conducted experimental investigations in the same configuration. Their results showed that AFC
125
successfully suppressed the flow separation with a lift increase and that results are independent on the
126
Reynolds and Mach numbers. Hue et al. [
43
] using RANS simulated constant and pulsed blowing
127
devices and observed gains up to 3% in the lift and the retard in separation due to the nacelle. Fluidic
128
actuators placed at a tail of an aircraft were simulated using unsteady RANS and validated by means
129
experimental results by Shmilovich et al. [
44
]. More recently, Andino et al. [
45
] tested fluidic actuators
130
in a generic tail at low speeds and demonstrated that a modest increase of the momentum coefficient
131
can result in important increments of the side force. Whalen et al. [
46
] presented wind-tunnel test
132
results of the AFC of the vertical tail of a Boeing 757 equipped with sweeping jet actuators; significant
133
increase in the side force at maximum rudder deflection of 30
◦
was observed. Other works involving
134
the use of sweeping jets can be found in Refs. [
47
,
48
]. The effectiveness of microjets at drag reduction
135
was experimentally studied by Aley et al. [
49
] in a simplified 2D wing; a significant wake velocity
136
deficit reduction and thus, drag was observed when using the microjets actuation.137
To finalise, we focus on the application of synthetic jets with zero net mass flux as a promising
138
technique for AFC of wings. In these devices the fluid necessary to alter the boundary layer is
139
intermittently injected through an orifice driven by the motion of a diaphragm located on a sealed
140
cavity below the surface [
50
]. Indeed, synthetic jets have been shown to succeed at reducing the fuel
141
burnt during the operations of take-off and landing [
51
]. In the context of synthetic jets for AFC, there
142
have been significant advances in the past years in airfoils (see for instance Refs. [
52
–
55
]). However,
143
whether they can be implement on a full aircraft is still subject of investigation. Recently, Jabbal et al.
144
[
56
] analysed different system architectures for AFC for real-size civil aircrafts in terms of efficiency,
145
power requirements and integration issues. They concluded that synthetic jets might be useful to
146
control separation in short-duration operations. Shmilovich and Yadlin [
57
] studied different AFC
147
strategies of a high-lift profile in the conditions of take-off and landing using RANS. Bauer et al. [
58
]
148
conducted experiments on a two element wing with unsteady AFC near the leading edge and showed
149
that stall can be delayed. Lin et al. [
59
] addressed different strategies in the flap of a high-lift profile
150
comprising steady suction and blowing, and periodic excitation of the boundary layer. Several of
151
these AFC strategies are planned to be tested experimentally by NASA for increasing lift-to-drag ratios
152
(
L/D
) in take-off configurations [
60
]. Although most of the numerical studies conducted so far have
153
Version January 4, 2022 submitted to Fluids 5 of 16
Figure 2.
JAXA Standard Model high-lift configuration at
Rec=
1.93
×
10
6
and
AoA =
21.57
o
with AFC.
Two-dimensional streamlines at different spanwise locations: baseline case vs different actuator-jet
angles
Φ
. (Top left) baseline; (top right)
Φ=
0
o
; (bottom left)
Φ=
60
o
; and (bottom right)
Φ=
45
o
.
Results extracted from Ref. [62], with permission of the publisher (IOP Publishing).
been performed using RANS, Jansen et al. [
61
] proved, by comparing experimental and numerical
154
simulations, that delayed detached eddy simulations are useful to analyse the effect of a synthetic jet
155
on the flow field of a tail at
Rec=
350, 000. Finally, Lehmkuhl et al. [
62
] have studied the aerodynamic
156
performance of active flow control on wings using synthetic jets with zero net-mass flow by means of
157
wall-modeled large-eddy simulations, see Figure 2. The performance of synthetic jets was evaluated
158
for the high-lift configuration of the JAXA Standard Model at realistic Reynolds numbers for landing
159
Rec=
1.96
×
10
6
. The results show that, at high angles of attack, the control successfully eliminates the
160
laminar/turbulent recirculations located downstream of the actuator, thus increasing the aerodynamic
161
performance.162
2. Turbulence-simulation approaches163
The use of computational fluid dynamics (CFD) for external-aerodynamic applications has been
164
a key tool for aircraft design in the modern aerospace industry [
63
–
65
]. CFD methodologies with
165
increasing functionality and performance have greatly improved our understanding and predictive
166
capabilities of complex flows. These improvements suggest that design of novel and highly reliable
167
control strategies via CFD may soon be a reality. The fully-virtual design of flow-control strategies is
168
expected to limit the number of required wind-tunnel tests, reducing both the turnover time and cost
169
of the design cycle [
66
,
67
]. However, flow predictions from the state-of-the-art CFD solvers are still
170
unable to comply with the stringent accuracy requirements and computational efficiency demanded by
171
the industry [
68
]. These limitations are imposed, largely, by the ubiquity of turbulence [
69
]. To tackle
172
current challenges and encourage further advances in CFD, simulation of an aircraft configuration
173
across the full flight envelope has been posed as one of the Grand Challenge Problems in the recent
174
NASA CFD Vision 2030 [68].175
From the early days of industrial CFD to present times, the treatment of turbulence has been mostly
176
based on closure models for the Reynolds-averaged Navier–Stokes (RANS) equations. The approach177
appears in different flavors: from pure RANS solutions to Hybrid methods such as Detached Eddy
178
Simulation and its variants [
70
,
71
]. In the latter, RANS is utilized close to the wall, whereas the outer
179
layer is modeled via eddy-resolving methodologies. Many RANS models (and their variants) have
180
been devised to overcome the limitations of their predecessors, usually by expanding and calibrating
181
its coefficients to account for missing physics. Despite the reliance of RANS-based approaches on
182
Version January 4, 2022 submitted to Fluids 6 of 16
tunable parameters and empirical correlations, they have dominated the CFD industry for external
183
aerodynamic applications, including commercial aviation [72].184
The sophistication of RANS closure models has increased over time [
71
]. Yet, no practical
185
model has emerged as a competent approach across the broad range of flow regimes of interest to
186
industry. The latter encompass separated flows, afterbodies, mean-flow three-dimensionality, shock
187
waves, aerodynamic noise, fine-scale mixing, laminar-to-turbulent transition, etc. In these scenarios,
188
RANS predictions tend to be inconsistent and unreliable, especially for geometries and conditions
189
representative of the flight envelope of commercial airplanes. An example of such deficiencies is the
190
prediction of the onset and extent of three-dimensional separated flow in wing-fuselage junctures, in
191
which RANS-based approaches have shown poor performance [
72
,
73
]. RANS accuracy is also known
192
to decline in aeroacoustic noise and vibration prediction for transonic airfoils [
72
]. Additional CFD
193
experience in aircrafts at high angles of attack has revealed that RANS-based solvers have difficulty
194
predicting maximum lift and the corresponding angle of attack along with the physical mechanisms
195
for stall. This was highlighted in the 3rd AIAA CFD High Lift Prediction Workshop [
74
], where RANS
196
solutions exhibited a significant scatter in lift, drag, and pitching moment near stall.197
Recently, large-eddy simulation (LES) has gained momentum as a tool for both research and
198
industrial applications. In LES, the large eddies containing most of the energy are directly resolved,
199
while the dissipative effect of the small scales is accounted for by a subgrid-scale (SGS) model.
200
Additionally, if the near-wall flow is also modeled (i.e., wall modeling) such that only the large-scale
201
motions in the outer region of the boundary layer are resolved, the grid-point requirements for
202
this wall-modeled LES (WMLES) scale at most linearly with increasing Reynolds number [
75
]. The
203
cost-efficiency of WMLES and its demonstrated predictive capabilities over the last decade, make this
204
approach a realistic contender to overcome the deficiencies of RANS-based methodologies.205
Several strategies for modeling the near-wall region in LES have appeared in the literature, and
206
comprehensive reviews can be found in Piomelli and Balaras
[76]
, Cabot and Moin
[77]
, Larsson et al.
207
[78]
, and the most recent review by Bose and Park
[79]
. Most wall models utilize as input the LES
208
solution at a given location in the LES domain, and return the wall heat and momentum fluxes needed
209
by the LES solver. Among the most widespread approaches are those computing the wall stress using
210
either the law of the wall [
80
–
82
] or simplified RANS equations [
83
–
90
], while recent advances in wall
211
modeling are rooted on mathematical and physical principles completely free of RANS empiricism
212
[91–93].213
Advances in machine learning and data science have also incited new efforts to complement
214
the existing turbulence-modeling approaches in the WMLES community. One of the first attempts
215
at using supervised machine learning for WMLES can be found in Yang et al.
[94]
, who proposed a
216
physics-informed neural-network (PINN) model to predict the wall stress in turbulent channel flows.
217
Recently, Lozano-Durán and Bae [
95
] formulated a wall model using building-block units (such as
218
turbulent channel flows, ducts, and separation bubbles), which provides a classification of the flow
219
and confidence in the prediction. The model was validated in a realistic aircraft with trailing-edge
220
separation. Radhakrishnan et al.
[96]
formulated a wall model using gradient-boosted decision trees
221
and predicted the wall-shear stress in a turbulent channel flow and a wall-mounted bump. Along these
222
lines, Eivazi et al. [97] have recently reported efforts towards RANS modelling by means of PINNs.223
According to NASA Vision 2030 report [
68
], hybrid RANS/LES and WMLES are identified
224
as the most viable approaches for predicting realistic flows at high Reynolds numbers in external
225
aerodynamics. As such, both hybrid RANS/LES and WMLES will be instrumental in the development
226
of control strategies for realistic external-aerodynamic applications, as shown in Figure 3.227
3. Data-driven methods for control and deep reinforcement learning228
As discussed above, the flow around wings is very complex and it is difficult to devise efficient
229
control strategies to optimize the aerodynamic efficiency, even having access to flow information in real
230
time as in the case of opposition control. One approach to obtain more efficient control strategies is to
231
Version January 4, 2022 submitted to Fluids 7 of 16
Figure 3.
Prediction of the wall-shear stress in the High-Lift Common Research Model at an angle of
attack of 21 degrees computed using wall-modeled large-eddy simulation with the Alya solver on a
grid with 40 million control elements.
formulate an optimization problem aimed at e.g. minimizing the drag or maximizing the aerodynamic
232
efficiency. There have been several data-driven approaches to do this for flow control, for instance
233
in the context of genetic programming [
98
]. Genetic programming (GP) is based on automatically
234
choosing the terms in a symbolic equation through evolution and selection of the best candidates, a
235
fact that ensures the interpretability of this method [
99
,
100
], although the formula obtained can be
236
deeply nested and complex. The GP approach has been successfully employed for control of external
237
flows by Li et al. [
101
] and Minelli et al. [
102
]. Another interesting data-driven approach is Bayesian
238
regression based on Gaussian processes [
103
], which was employed by Morita et al. [
104
] in CFD
239
optimization, and by Mahfoze et al. [
105
] to identify the best combination of control-region length and
240
blowing amplitude to maximize the energy savings, also including intermittent control regions. Note
241
that these authors also took into account the data by Kornilov and Boiko [
106
] to formulate a more
242
realistic estimate of the power consumption by blowing, and they reported a net-energy saving of
243
around 5%. It is interesting to note that other data-driven methods may help to model the near-wall
244
region and consequently may provide novel venues for improved flow control [107–111].245
One very promising data-driven approach to flow control is deep reinforcement learning (DRL),
246
which we will focus on in the following. In DRL, an agent (usually built based on a neural network, NN)
247
interacts with an environment (the flow) in a closed loop. At each time
t
, the agent receives a partial
248
observation of the environment
ot
used to choose an action
at
, which will influence the evolution of the
249
environment. The agent periodically receives a reward
rt
, which indicates the quality of those actions
250
under a certain norm. The goal of DRL is to find an optimal decision policy πfrom with the action is251
derived, i.e.
at=π(ot)
, such that the cumulative reward is maximized. This process is summarized
252
in Figure 4, and the goal of the DRL algorithm is to learn by interacting with the environment by
253
gathering experience [
112
]. A pioneering work in this direction in the context of instability control in
254
fluid mechanics was conducted by Rabault et al. [113], who used DRL to optimize the actuation from255
two jets on a two-dimensional cylinder flow. This resulted in significant drag-reduction rates using
256
synthetic control jets blowing with a very low mass-flow rate intensity (typically a fraction of percent
257
of the incoming mass flow rate intersecting the cylinder). We want to highlight that DRL is a very
258
Version January 4, 2022 submitted to Fluids 8 of 16
Figure 4.
Schematic representation of the deep-reinforcement-learning process employed for flow
control. The objective is to find the optimal decision policy
π
such that the cumulative reward is
maximized.
promising approach to discover novel and potentially more efficient control strategies going beyond
259
classical control, since i) it does not make any assumptions on the properties of the system, except for
260
the ability to establish a close-loop control and to extract a reward signal, ii) it takes advantage of the261
efficiency of NNs at representing complex, nonlinear functions following their universal-approximator
262
property [114].263
The work by Rabault et al. [
113
] employed the proximal-policy-optimization (PPO) algorithm [
115
],
264
which is an Actor-Critic policy gradient algorithm. The PPO algorithm is simpler and faster than other
265
similar techniques, such as the trust region policy optimization (TRPO) methods [
115
], and it requires
266
relatively little hyper-parameter tuning. Furthermore, it is more suitable for continuous control than
267
the deep-Q-network (DQN) learning [
116
], as well as its variations [
117
]. Offering a detailed overview
268
of the PPO algorithm is beyond the scope of this review, and for a detailed discussion about the PPO
269
algorithm, the reader is referred to either the initial PPO paper [
115
], or to a fluid-mechanics focused
270
review of the DRL and PPO method [
118
,
119
]. However, the main lines of the PPO algorithm are as
271
follows. The PPO method is episode-based, i.e., it learns from performing active control for a limited
272
amount of time before analyzing the obtained results and continuing with the learning process in a
273
new episode. The learning problem is aimed at iteratively training (i.e., finding the weights of) the
274
policy network. Denoting the set of weights of the policy NN by
Θ
, the aim is, therefore, to maximize
275
the long-term discounted reward function
R(t) = ∑tγtrt
, where
γ
is a discount factor (usually in the
276
range [0.95 −0.99]), formulated as finding:277
Rmax =max
Θ
E"H
∑
t=0γtrt|πΘ#, (2)
where
πΘ
is the policy function described by the neural network with weights
Θ
, and
st
is the (hidden)
278
state of the system. In the present context,
st
would correspond to the complete flow information,
279
whereas the limited observations
ot
would be obtained from sensors. This maximization problem
280
is solved by means of gradient descent performed on the weights
Θ
of the network, following
281
experimental sampling of the system through interaction with the environment.282
Following the initial work controlling the vortex shedding in a cylinder wake, a number of
283
further refinements and applications have proven the potential of the method for the control of
284
flow instabilities. Bucci et al.
[120]
successfully applied DRL to control chaotic systems, such as the
285
Kuramoto–Sivashinsky equation. Paris et al.
[121]
investigated how sensors providing an overview of
286
the state of the system to the DRL agent can be placed optimally. Beintema et al.
[122]
demonstrated
287
efficient control of the Rayleigh Benard instability in a 2D channel. Tang et al.
[123]
proved through
288
numerical simulations that DRL can be able to perform robust control over a range of inflow conditions.
289
Xu et al.
[124]
investigated in a simulation how small counter-rotating cylinders can be used to reduce
290
the drag behind a cylinder, while Fan et al.
[125]
provided an experimental demonstration of the
291
technique. Finally, Ren et al.
[126]
pushed the value of the Reynolds number to a weakly turbulent
292
regime and demonstrated that DRL can control fluid motion in the turbulent regime.293
Version January 4, 2022 submitted to Fluids 9 of 16
Figure 5.
Illustration of the self-collaborative approach demonstrated by Belus et al.
[128]
for controlling
systems with several similar control outputs. In this method, clones of the DRL agent are used on each
of the output and share their experience samples and training. This allows to learn control laws in
constant time, independently of the number of outputs used. This figure is reproduced with minor
changes from Belus et al. [128].
While research articles have mostly focused on relatively simple flow configurations so far, as
294
these are the easiest to tackle computationally both for the CFD and the DRL agent, there are a
295
number of possible refinements in the use of the PPO algorithm that make it a promising method
296
also for controlling more complex, 3D cases. Firstly, the PPO algorithm is able to sample data
297
from several independent environments when performing learning. Therefore, one can effectively
298
parallelize the DRL training by using many CFD simulations running in parallel, as was presented by
299
Rabault and Kuhnle [
127
]. This allows to drastically accelerate the training ([
127
] reports speed ups of
300
up to 20, though the more complex the system to control, the higher the speed up attainable using this
301
technique). This allows to perform reasonably fast PPO training, even in cases when the underlying
302
environment is difficult to speed up. For example, in the case of CFD-based environment, this allows
303
to scale the training to a number of compute cores
N×M
, where
N
is the number of simulations run
304
in parallel, and
M
is the maximum optimal parallelization of the CFD simulation itself. Secondly, it
305
is possible to formulate the learning problem in such a way as to take advantage of invariants in the
306
physical system that is undergoing control, as was illustrated by Belus et al.
[128]
. In their work, Belus
307
et al.
[128]
formulated the learning problem as a self-collaborative interaction between several clones
308
of the same environment, as is visible in Figure 5. This, in turn, allows the DRL agent to combine
309
the information obtained at many locations that follow the same physical rules into a single policy.
310
Belus et al.
[128]
argue based on theoretical considerations and prove empirically that, without such
311
a technique, performing learning on a system with
No
outputs has a cost that scales as
CNo
, when
C312
is the cost of training for a single output. This is prohibitively expensive as the number of outputs
313
is increased. Belus et al.
[128]
then demonstrated empirically that, by contrast, using the approach
314
presented in Figure 5allows to perform training at a constant cost, independently of the number of
315
outputs, as long as the control law to learn is similar at all outputs. This is a critical enabling factor
316
for the application of DRL to realistic configurations, where many similar control outputs will be
317
distributed across the physical system to control.318
4. Conclusions and outlook319
Machine Learning-based control methods are an exciting set of techniques that are receiving
320
considerable attention recently for performing active flow control. This spike in interest follows both321
increases in computational power and the development of effective algorithms that can learn effectively
322
through direct interaction with black-box, complex systems. These ML methods follow a completely
323
Version January 4, 2022 submitted to Fluids 10 of 16
different approach compared with how flow control strategies have usually been designed. Instead
324
of performing a local analysis of the flow properties by considering the flow equations and using
325
advanced mathematical and analytical tools to find optimal perturbations, ML techniques discover
326
control strategy through a trial-and-error approach. There are a number of promising methods that
327
belong to the ML family of control algorithms, including for example Genetic Programming (GP) and
328
Deep Reinforcement Learning (DRL). In this review, we focused on DRL methods in particular, and
329
we discussed how recent works indicate that it can be efficiently used for controlling large, complex,
330
non-linear systems arising from control tasks in fluid mechanics.331
The efficiency of DRL has been demonstrated in a number of active flow control situations
332
so far, and the fluid mechanics community is progressively tackling more and more complex flow
333
configurations. In particular, recent works are pushing the use of DRL for flow control into intermediate
334
Re values, leading to more non-linear and more complex flows, so far successfully. The next steps will
335
be to demonstrate DRL control of complex 3D CFD simulations, and to further increase Re to reach
336
fully turbulent conditions. While this will pose new challenges to the DRL method due to the inherent
337
increase in complexity compared with the configurations studied so far, a number of preliminary
338
works indicate that DRL is well adapted to controlling complex systems with a large number of control
339
locations, and that the inherent parallelism present in the DRL experience sampling process can offer
340
large speed ups on complex dynamical systems.341
This push to more complex systems represents not only a scientific, but also a technical endeavour.
342
Indeed, applying DRL to 3D flow control at moderate to high Re will pose a number of technical
343
challenges regarding the amount of CFD computational power required, the ability to handle large
344
amounts of data, and the coupling of CFD and DRL codes that were designed independently of each345
other at a time when the ability to couple them was not yet foreseen. All theses aspects put tough
346
requirements on the level of both expert knowledge (few people are expert in both DRL and large
347
scale CFD) and general technical expertise (combining several different complex software stacks into a
348
single system, and deploying this in HPC environments). In our opinion, these technical challenges,
349
rather than fundamental issues, are presently the main limiting factor for applying ML control to active
350
flow control. A possible way out of this challenge is to follow the example set by the ML community
351
and adopt a resolute open source release policy of codes, scripts, tutorials, and trained networks, to
352
reduce the barrier to entry for new groups joining in this research direction.353
Author Contributions:
All authors contributed towards the ideation, writing of the original draft and editing of
354
the manuscript. All authors have read and agreed to the published version of the manuscript.355
Acknowledgments:
RV acknowledges the financial support from the Swedish Research Council (VR). OL is
356
partially financed by a Ramón y Cajal postdoctoral contract by the Ministerio de Economía y Competitividad,
357
Secretaría de Estado de Investigación, Desarrollo e Innovación, Spain (RYC2018-025949-I).358
Conflicts of Interest: The authors declare no conflict of interest.359
References360
1.
Bouwer, J. Will airline hubs recover from COVID-19? Available at: https://www.mckinsey.com/industries/
361
travel-logistics-and-transport-infrastructure/our-insights/ will-airline-hubs- recover-from-covid-19# 2020.362
2.
Liu, J.et al.. Will the aviation industry have a bright future after the COVID-19 outbreak? Evidence from
363
Chinese airport shipping sector. Journal of Risk and Financial Management 2020,13, 276.364
3.
Owen, B.; Lee, D.S.; Lim, L. Flying into the future: Aviation emissions scenarios to 2050. Environ. Sci.
365
Technol. 2010,44, 2255–2260.366
4.
Kroo, I. Drag due to lift: Concepts for prediction and reduction. Annu. Rev. Fluid Mech.
2001
,33, 587–617.
367
5.
UN General Assembly. Transforming our world: the 2030 Agenda for Sustainable Development. Resolut.
368
A RES 70 1 2015,25, 1–35.369
6.
Vinuesa, R.; Azizpour, H.; Leite, I.; Balaam, M.; Dignum, V.; Domisch, S.; Felländer, A.; Langhans, S.D.;
370
Tegmark, M.; Fuso Nerini, F. The role of artificial intelligence in achieving the Sustainable Development
371
Goals. Nat. Commun. 2020,11, 233.372
Version January 4, 2022 submitted to Fluids 11 of 16
7.
Gupta, S.; Langhans, S.D.; Domisch, S.; Fuso-Nerini, F.; Felländer, A.; Battaglini, M.; Tegmark, M.; Vinuesa,
373
R. Assessing whether artificial intelligence is an enabler or an inhibitor of sustainability at indicator level.
374
Transp. Eng. 2021,4, 100064.375
8.
Fahland, G.; Stroh, A.; Frohnapfel, B.; Atzori, M.; Vinuesa, R.; Schlatter, P.; Gatti, D. Investigation of
376
blowing and suction for turbulent flow control on airfoils. AIAA J., https://doi.org/10.2514/1.J060211 2021.377
9.
Bechert, D.W.; Bartenwerfer, M. The viscous flow on surfaces with longitudinal ribs. J. Fluid Mech.
1989
,
378
206, 105–112.379
10. Viswanath, P.R. Aircraft viscous drag reduction using riblets. Prog. Aerosp. Sci. 2002,38, 571–600.380
11.
Gad-el Hak, M. Flow control – Passive, active and reactive flow management. Cambridge University Press,
381
Cambridge. 2000.382
12. Choi, H.; Jeon, W.P.; Kim, J. Control of flow over a bluff body. Annu. Rev. Fluid Mech. 2008,40, 113–139.383
13.
Quadrio, M. Drag reduction in turbulent boundary layers by in-plane wall motion. Philos. Trans. Royal Soc.
384
A2011,369, 1428–1442.385
14.
Choi, H.; P., M.; Kim, J. Active turbulence control for drag reduction in wall-bounded flows. J. Fluid Mech.
386
1994,262, 75–110.387
15.
Stroh, A.; Frohnapfel, B.; Schlatter, P.; Hasegawa, Y. A comparison of opposition control in turbulent
388
boundary layer and turbulent channel flow. Phys. Fluids 2015,27, 075101.389
16.
Stroh, A.; Frohnapfel, B.; Schlatter, P.; Hasegawa, Y. Status and perspectives of laminar flow. Aeronaut. J.
390
2005,109, 639–644.391
17.
Wood, R.M. Impact of advanced aerodynamic technology on transportation energy consumption. SAE
392
Technical Papers 2004,113, 854–874.393
18.
Abbas, A.; De Vicente, J.; Valero, E. Aerodynamic technologies to improve aircraft performance. Aerosp.
394
Sci. Technol. 2013,28, 100–132.395
19.
Hwang, D. A proof of concept experiment for reducing skin friction by using a micro-blowing technique.
396
35th Aerospace Sciences Meeting and Exhibit 1997.397
20.
Hwang, D. Review of research into the concept of the microblowing technique for turbulent skin friction
398
reduction. Prog. Aerosp. Sci. 2004,40, 559–575.399
21.
Kornilov, V.I. Current state and prospects of researches on the control of turbulent boundary layer by air
400
blowing. Prog. Aerosp. Sci. 2015,76, 1–23.401
22.
Park, J.; Choi, H. Effects of uniform blowing or suction from a spanwise slot on a turbulent boundary layer
402
flow. Phys. Fluids 1999,11, 3095–3105.403
23.
Kametani, Y.; Fukagata, K. Direct numerical simulation of spatially developing turbulent boundary layers
404
with uniform blowing or suction. J. Fluid Mech. 2011,681, 154–172.405
24.
Kametani, Y.; Fukagata, K.; Örlü, R.; Schlatter, P. Effect of uniform blowing/suction in a turbulent boundary
406
layer at moderate Reynolds number. Int. J. Heat Fluid Flow 2015,55, 132–142.407
25.
Noorani, A.; Vinuesa, R.; Brandt, L.; Schlatter, P. Aspect ratio effect on particle transport in turbulent duct
408
flows. Phys. Fluids 2016,28, 115103.409
26.
Bobke, A.; Vinuesa, R.; Örlü, R.; Schlatter, P. History effects and near equilibrium in
410
adverse-pressure-gradient turbulent boundary layers. J. Fluid Mech. 2017,820, 667–692.411
27.
Vinuesa, R.; Negi, P.S.; Atzori, M.; Hanifi, A.; Henningson, D.S.; Schlatter, P. Turbulent boundary layers
412
around wing sections up to Rec=1, 000, 000. Int. J. Heat Fluid Flow 2018,72, 86–99.413
28.
Tanarro, A.; Vinuesa, R.; Schlatter, P. Effect of adverse pressure gradients on turbulent wing boundary
414
layers. J. Fluid Mech. 2020,883, A8.415
29.
Eto, K.; Kondo, Y.; Fukagata, K.; Tokugawa, N. Assessment of friction drag reduction on a clark-Y airfoil
416
by uniform blowing. AIAA J. 2019,57, 2774–2782.417
30.
Kornilov, V.I.; Kavun, I.N.; Popkov, A.N. Modification of turbulent airfoil section flow using a combined
418
control action. Thermophys. Aeromechanics 2019,26, 165–178.419
31.
Vinuesa, R.; Schlatter, P. Skin-friction control of the flow around a wing section through uniform blowing.
420
Proc. European Drag Reduction and Flow Control Meeting (EDRFCM) 2017.421
32.
Atzori, M.; Vinuesa, R.; Fahland, G.; Stroh, A.; Gatti, D.; Frohnapfel, B.; Schlatter, P. Aerodynamic effects of
422
uniform blowing and suction on a NACA4412 airfoil. Flow Turbul. Combust. 2020,105, 735–759.423
33.
Albers, M.; Meysonnat, P.S.; Schröder, W. Actively reduced airfoil drag by transversal surface waves. Flow
424
Turbul. Combust. 2019,102, 865–886.425
Version January 4, 2022 submitted to Fluids 12 of 16
34. Jeong, J.; Hussain, F. On the identification of a vortex. J. Fluid Mech. 1995,285, 69–94.426
35.
Hosseini, S.M.; Vinuesa, R.; Schlatter, P.; Hanifi, A.; Henningson, D.S. Direct numerical simulation of the
427
flow around a wing section at moderate Reynolds number. Int. J. Heat Fluid Flow 2016,61, 117–128.428
36.
Vinuesa, R.; Hosseini, S.M.; Hanifi, A.; Henningson, D.S.; Schlatter, P. Pressure-gradient turbulent boundary
429
layers developing around a wing section. Flow Turbul. Combust. 2017,99, 613–641.430
37.
Batikh, A.; Baldas, L.; Colin, S. Application of Active Flow Control on Aircrafts - State of the Art.
431
International Workshop on Aircraft System Technologies 2017, pp. 1–10.432
38.
Khodadoust, A.; Washburn, A. Active control of flow separation on a high-lift system with slotted flap at
433
high Reynolds number. Collection of Technical Papers - AIAA Applied Aerodynamics Conference, 2007,
434
Vol. 3, pp. 1645–1669. doi:10.2514/6.2007-4424.435
39.
Khün, T.; Ciobaca, V.; Rudnik, R.; Gölling, B.; Breitenstein, W. Active flow separation control on a high-lift
436
wing-body configuration part 1: Baseline flow and constant blowing. 29th AIAA Applied Aerodynamics
437
Conference 2011, 2011, pp. 1–11. doi:10.2514/6.2011-3168.438
40.
Radespiel, R.; Burnazzi, M.; Casper, M.; Scholz, P. Active flow control for high lift with steady blowing.
439
Aeronautical Journal 2016,120, 171–200. doi:10.1017/aer.2015.7.440
41.
Fricke, S.; Ciobaca, V.; Kröhnert, A.; Wild, J.; Blesbois, O. Active Flow Control Applied at the Engine-Wing
441
Junction. 5th CEAS Air & Aerospace conference, 2015, number 249, pp. 1–12.442
42.
Schloesser, P.; Soudakov, V.; Bauer, M.; Wild, J. Active separation control at the pylon-wing junction of a
443
real-scale model. AIAA Journal 2019,57, 132–141. doi:10.2514/1.J057345.444
43.
Hue, D.; François, C.; Dandois, J.; Gebhardt, A. Simulations of an aircraft with constant and pulsed
445
blowing flow control at the engine/wing junction. Aerospace Science and Technology
2017
,69, 659–673.
446
doi:10.1016/j.ast.2017.07.031.447
44.
Shmilovich, A.; Yadlin, Y.; Whalen, E.A. Active flow control computations: From a single actuator to a
448
complete airplane. AIAA Journal 2018,56, 4730–4740. doi:10.2514/1.J056307.449
45.
Andino, M.Y.; Lin, J.C.; Roman, S.; Graff, E.C.; Gharib, M.; Whalen, E.A.; Wygnanski, I.J. Active Flow
450
Control on Vertical Tail Models. AIAA Journal 2019,57, 3322–3338. doi:10.2514/1.J057876.451
46.
Whalen, E.A.; Lacy, D.; Lin, J.C.; Andino, M.Y.; Washburn, A.E.; Graff, E.; Wygnanski, I.J. Performance
452
enhancement of a full-scale vertical tail model equipped with active flow control. 53rd AIAA Aerospace
453
Sciences Meeting 2015. doi:10.2514/6.2015-0784.454
47.
Kara, K.; Kim, D.; Morris, P.J. Flow-separation control using sweeping jet actuator. AIAA Journal
2018
,
455
56, 4604–4613. doi:10.2514/1.J056715.456
48.
Chen, S.; Li, W.; Meng, Q.; Zhou, Z.; Wang, S. Effects of a sweeping jet actuator on aerodynamic
457
performance in a linear turbine cascade with tip clearance. Proceedings of the Institution of Mechanical
458
Engineers, Part G: Journal of Aerospace Engineering 2019,233, 4468–4481. doi:10.1177/0954410018823606.459
49.
Aley, K.S.; Guha, T.K.; Kumar, R. Active Flow Control of a High-Lift Supercritical Airfoil with Microjet
460
Actuators. AIAA Journal 2020, pp. 1–17. doi:10.2514/1.j058939.461
50.
Glezer, A. Some aspects of aerodynamic flow control using synthetic-jet actuation. Philosophical
462
Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
2011
,369, 1476–1494.
463
doi:10.1098/rsta.2010.0374.464
51.
Hartwich, P.M.; Shmilovich, A.; Lacy, D.S.; Dickey, E.D.; Scalafani, A.J.; Sundaram, P.; Yadlin, Y. Refined
465
AFC-Enabled High-Lift System Integration Study. Technical Report CR-2016-219170, 2016.466
52.
Mccormick, D.C. Boundary Layer Separation control with Directed Synthetic Jets. 38th Aerospace Sciences
467
Meeting & Exhibit, 2000, number AIAA 2000-0519. doi:10.2514/6.2000-519.468
53.
Amitay, M.; Glezer, a. Role of actuation frequency in controlled flow reattachment over a stalled airfoil.
469
AIAA Journal 2002,40, 209–216. doi:10.2514/3.15052.470
54.
Gilarranz, J.L.; Traub, L.W.; Rediniotis, O.K. A New Class of Synthetic Jet Actuators—Part I:
471
Design, Fabrication and Bench Top Characterization. Journal of Fluids Engineering
2005
,127, 367.
472
doi:10.1115/1.1839931.473
55.
You, D. and Moin, P. Active control of flow separation over an airfoil using synthetic jets. Journal of Fluids
474
and Structures 2008,24, 1349–1357. doi:10.1016/j.jfluidstructs.2008.06.017.475
56.
Jabbal, M.; Liddle, S.C.; Crowther, W.J. Active flow control systems architectures for civil transport aircraft.
476
Journal of Aircraft 2010,47, 1966–1981. doi:10.2514/1.C000237.477
Version January 4, 2022 submitted to Fluids 13 of 16
57.
Shmilovich, A.; Yadlin, Y. Active Flow Control for Practical High-Lift Systems. Journal of Aircraft
2009
, pp.
478
1354–1364.479
58.
Bauer, M.; Lohse, J.; Haucke, F.; Nitsche, W. High-Lift Performance Investigation of a Two-Element
480
Configuration with a Two-Stage Actuator System. AIAA Journal
2014
,52, 1307–1313. doi:10.2514/1.j052639.
481
59.
Lin, J.C.; Pack Melton, L.G.; Viken, S.; Andino, M.Y.; Koklu, M.; Hannon, J.; Vatsa, V.N. High Lift Common
482
Research Model for Wind Tunnel Testing: An Active Flow Control Perspective. 55th AIAA Aerospace
483
Sciences Meeting, 2017. doi:10.2514/6.2017-0319.484
60.
Lin, J.C.; Melton, L.G.P.; Hannon, J.; Andino, M.; Koklu, M.; Paschal, K.; Vatsa, V.N. Wind Tunnel
485
Testing of Active Flow Control on the High Lift Common Research Model. AIAA Aviation 2019 Forum.
486
doi:10.2514/6.2019-3723.487
61.
Jansen, K.E.; Rasquin, M.; Farnsworth, J.A.; Rathay, N.; Monastero, M.C.; Amitay, M. Interaction
488
of a Synthetic Jet with Separated Flow over a Vertical Tail. AIAA Journal
2018
,56, 2653–2668.
489
doi:10.2514/1.j056751.490
62.
Lehmkuhl, O.; Lozano-Durán, A.; Rodriguez, I. Active flow control for external aerodynamics: from
491
micro air vehicles to a full aircraft in stall. Journal of Physics: Conference Series
2020
,1522, 012017.
492
doi:10.1088/1742-6596/1522/1/012017.493
63.
Johnson, F.T.; Tinoco, E.N.; Yu, N.J. Thirty years of development and application of CFD at Boeing
494
Commercial Airplanes, Seattle. Computers & Fluids 2005,34, 1115–1151.495
64.
Abbas-Bayoumi, A.; Becker, K. An industrial view on numerical simulation for aircraft aerodynamic
496
design. Journal of Mathematics in Industry 2011,1, 1–14.497
65.
Spalart, P.R.; Venkatakrishnan, V. On the role and challenges of CFD in the aerospace industry. The
498
Aeronautical Journal 2016,120, 209–232. doi:10.1017/aer.2015.10.499
66.
Clark, A.M.; Slotnick, J.P.; Taylor, N.J.; Rumsey, C.L. Requirements and challenges for CFD
500
validation within the High-Lift Common Research Model ecosystem. AIAA Aviation Forum, 2020,
501
[https://arc.aiaa.org/doi/pdf/10.2514/6.2020-2772]. doi:10.2514/6.2020-2772.502
67.
Recommended Practice: When Flight Modelling Is Used to Reduce Flight Testing Supporting
503
Aircraft Certification (AIAA R-154-2021). In Recommended Practice: When Flight Modelling
504
Is Used to Reduce Flight Testing Supporting Aircraft Certification (AIAA R-154-2021); 2021;
505
[https://arc.aiaa.org/doi/pdf/10.2514/4.106231.001]. doi:10.2514/4.106231.001.506
68.
Slotnick, J.; Khodadoust, A.; Alonso, J.; Darmofal, D.; Gropp, W.; Lurie, E.; Mavriplis, D. CFD Vision 2030
507
Study: A Path to Revolutionary Computational Aerosciences; Tech. Rep. CR–2014-218178, NASA, 2014.508
69.
Lozano-Durán, A.; Bose, S.T.; Moin, P. Performance of Wall-Modeled LES with Boundary-LayerConforming
509
Grids for External Aerodynamics. AIAA J. 2021, p. J061041.510
70.
Spalart, P.R.; Jou, W.H.; Strelets, M.; Allmaras, S.R. Comments on the feasibility of LES for wings, and on a
511
hybrid RANS/LES approach. Advances in DNS/LES 1997,1, 4–8.512
71. Spalart, P.R. Detached-eddy simulation. Annu. Rev. Fluid Mech. 2009,41, 181–202.513
72.
Casey, M.; Wintergerste, T.; European Research Community on Flow, T.; Combustion.. ERCOFTAC best
514
practice guidelines: ERCOFTAC special interest group on “quality and trust in industrial CFD”; ERCOFTAC,
515
2000.516
73.
Rumsey, C.L.; Slotnick, J.P.; Sclafani, A.J. Overview and Summary of the Third AIAA High Lift Prediction
517
Workshop. J. Aircr. 2019,56, 621–644. doi:10.2514/1.C034940.518
74.
Rumsey, C.L.; Carlson, J.; Ahmad, N. FUN3D Juncture Flow Computations Compared with Experimental
519
Data. AIAA Scitech 2019 Forum, 2019. doi:10.2514/6.2019-0079.520
75.
Choi, H.; Moin, P. Grid-point requirements for large eddy simulation: Chapman’s estimates revisited.
521
Phys. Fluids 2012,24, 011702.522
76.
Piomelli, U.; Balaras, E. Wall-layer models for large-eddy simulations. Annu. Rev. Fluid Mech.
2002
,
523
34, 349–374. doi:10.1146/annurev.fluid.34.082901.144919.524
77.
Cabot, W.H.; Moin, P. Approximate wall boundary conditions in the large-eddy simulation of high
525
Reynolds number flow. Flow Turbul. Combust. 2000,63, 269–291. doi:10.1023/A:1009958917113.526
78.
Larsson, J.; Kawai, S.; Bodart, J.; Bermejo-Moreno, I. Large eddy simulation with modeled wall-stress:
527
recent progress and future directions. Mech. Eng. Rev. 2016,3, 1–23. doi:10.1299/mer.15-00418.528
79.
Bose, S.T.; Park, G.I. Wall-Modeled LES for Complex Turbulent Flows. Ann. Rev. Fluid Mech.
2018
,
529
50, 535–561. doi:10.1146/annurev-fluid-122316-045241.530
Version January 4, 2022 submitted to Fluids 14 of 16
80.
Deardorff, J. A numerical study of three-dimensional turbulent channel flow at large Reynolds numbers. J.
531
Fluid Mech. 1970,41, 453–480.532
81.
Schumann, U. Subgrid scale model for finite difference simulations of turbulent flows in plane channels
533
and annuli. J. Comp. Phys. 1975,18, 376–404. doi:10.1016/0021-9991(75)90093-5.534
82.
Piomelli, U.; Ferziger, J.; Moin, P.; Kim, J. New approximate boundary conditions for large eddy simulations
535
of wall-bounded flows. Phys. Fluids 1989,1, 1061–1068. doi:10.1063/1.857397.536
83.
Balaras, E.; Benocci, C.; Piomelli, U. Two-layer approximate boundary conditions for large-eddy
537
simulations. AIAA J. 1996,34, 1111–1119. doi:10.2514/3.13200.538
84.
Wang, M.; Moin, P. Dynamic wall modeling for large-eddy simulation of complex turbulent flows. Phys.
539
Fluids 2002,14, 2043–2051.540
85.
Chung, D.; Pullin, D.I. Large-eddy simulation and wall modelling of turbulent channel flow. J. Fluid Mech.
541
2009,631, 281–309. doi:10.1017/S0022112009006867.542
86.
Bodart, J.; Larsson, J. Wall-modeled large eddy simulation in complex geometries with application to
543
high-lift devices. Center for Turbulence Research - Annual Research Briefs 2011, pp. 37–48.544
87.
Kawai, S.; Larsson, J. Dynamic non-equilibrium wall-modeling for large eddy simulation at high Reynolds
545
numbers. Phys. Fluids 2013,25, 015105.546
88.
Bermejo-Moreno, I.; Campo, L.; Larsson, J.; Bodart, J.; Helmer, D.; Eaton, J.K. Confinement effects in shock
547
wave/turbulent boundary layer interactions through wall-modelled large-eddy simulations. J. Fluid Mech.
548
2014,758, 5–62. doi:10.1017/jfm.2014.505.549
89.
Park, G.I.; Moin, P. An improved dynamic non-equilibrium wall-model for large eddy simulation. Phys.
550
Fluids 2014,26, 015108. doi:10.1063/1.4861069.551
90.
Yang, X.I.A.; Sadique, J.; Mittal, R.; Meneveau, C. Integral wall model for large eddy simulations of
552
wall-bounded turbulent flows. Phys. Fluids 2015,27, 025112. doi:10.1063/1.4908072.553
91.
Bose, S.T.; Moin, P. A dynamic slip boundary condition for wall-modeled large-eddy simulation. Phys.
554
Fluids 2014,26, 015104. doi:10.1063/1.4849535.555
92.
Lozano-Durán, A.; Bae, H.; Bose, S.; Moin, P. Dynamic wall models for the slip boundary condition. Center
556
for Turbulence Research - Annual Research Briefs 2017, pp. 229–242.557
93.
Bae, H.J.; Lozano-Durán, A.; Bose, S.T.; Moin, P. Dynamic slip wall model for large-eddy simulation. J.
558
Fluid Mech. 2019,859, 400–432. doi:10.1017/jfm.2018.838.559
94.
Yang, X.I.A.; Zafar, S.; Wang, J.X.; Xiao, H. Predictive large-eddy-simulation wall modeling via
560
physics-informed neural networks. Phys. Rev. Fluids
2019
,4, 034602. doi:10.1103/PhysRevFluids.4.034602.
561
95.
Lozano-Durán, A.; Bae, H.J. Self-critical machine-learning wall-modeled LES for external aerodynamics.
562
Center for Turbulence Research - Annual Research Briefs 2020, pp. 197–210.563
96.
Radhakrishnan, S.; Gyamfi, L.A.; Miro, A.; Font, B.; J., C.; Lehmkuhl, O. A data-driven wall-shear stress
564
model for LES using gradient boosted decision trees. International Conference on High Performance Computing
565
2021, pp. 105–121.566
97.
Eivazi, H.; Tahani, M.; Schlatter, P.; Vinuesa, R. Physics-informed neural networks for solving
567
Reynolds-averaged Navier–Stokes equations. Preprint arXiv:2107.10711 2021.568
98.
Koza, J.R. Genetic Programming: On the Programming of Computers by Means of Natural Selection; MIT Press:
569
Cambridge, MA, USA, 1992.570
99.
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use
571
interpretable models instead. Nat. Mach. Intell. 2019,1, 206–215.572
100.
Vinuesa, R.; Sirmacek, B. Interpretable deep-learning models to help achieve the Sustainable Development
573
Goals. Nat. Mach. Intell. 2021,3, 926.574
101.
Li, R.; Noack, B.R.; Cordier, L.; Borée, J.; Harambat, F. Drag reduction of a car model by linear genetic
575
programming control. Exp. Fluids 2017,58, 103.576
102.
Minelli, G.; Dong, T.; Noack, B.R.; Krajnovi´c, S. Upstream actuation for bluff-body wake control driven by
577
a genetically inspired optimization. J. Fluid Mech. 2020,893, A1.578
103.
Rasmussen, C.E. Gaussian processes for machine learning. Journal fur Urologie und Urogynakologie. Berlin:
579
Springer International Publishing 2003,7.580
104.
Morita, Y.; Rezaeiravesh, S.; Tabatabaei, N.; Vinuesa, R.; Fukagata, K.; Schlatter, P. Applying Bayesian
581
optimization with Gaussian-process regression to Computational Fluid Dynamics problems. Preprint
582
arXiv:2101.09985 2021.583
Version January 4, 2022 submitted to Fluids 15 of 16
105.
Mahfoze, O.A.; Moody, A.; Wynn, A.; Whalley, R.D.; Laizet, S. Reducing the skin-friction drag of a
584
turbulent boundary-layer flow with low-amplitude wall-normal blowing within a Bayesian optimization
585
framework. Phys. Rev. Fluids 2019,4, 094601.586
106.
Kornilov, V.I.; Boiko, A.V. Efficiency of air microblowing through microperforated wall for flat plate drag
587
reduction. AIAA J. 2012,50, 724–732.588
107.
Guastoni, L.; Güemes, A.; Ianiro, A.; Discetti, S.; Schlatter, P.; Azizpour, H.; Vinuesa, R.
589
Convolutional-network models to predict wall-bounded turbulence from wall quantities. J. Fluid Mech.
590
2021,928, A27.591
108.
Güemes, A.; Discetti, S.; Ianiro, A.; Sirmacek, B.; Azizpour, H.; Vinuesa, R. From coarse wall measurements
592
to turbulent velocity fields through deep learning. Phys. Fluids 2021,3, 075121.593
109.
Abreu, L.I.; Cavalieri, A.V.G.; Schlatter, P.; Vinuesa, R.; Henningson, D.S. Spectral proper orthogonal
594
decomposition and resolvent analysis of near-wall coherent structures in turbulent pipe flows. J. Fluid
595
Mech. 2020,900, A11.596
110.
Abreu, L.I.; Tanarro, A.; Cavalieri, A.V.; Schlatter, P.; Vinuesa, R.; Hanifi, A.; Henningson, D.S.
597
Spanwise-coherent hydrodynamic waves around flat plates and airfoils. J. Fluid Mech. 2021,927, A1.598
111.
Eivazi, H.; Le Clainche, S.; Hoyas, S.; Vinuesa, R. Towards extraction of orthogonal and parsimonious
599
non-linear modes from turbulent flows. Preprint arXiv:2109.01514 2021.600
112.
François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M.G. An Introduction to Deep Reinforcement
601
Learning. Foundations and Trends in Machine Learning, II 2018,3, 1–140.602
113.
Rabault, J.; Kuchta, M.; Jensen, A.; Réglade, U.; Cerardi, N. Artificial neural networks trained through deep
603
reinforcement learning discover control strategies for active flow control. J. Fluid Mech.
2019
,865, 281–302.
604
114.
Hornik, K.; Stinchcombe, M.; White, H. Multilayer feedforward networks are universal approximators.
605
Neural networks 1989,2, 359–366.606
115.
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms.
607
Preprint ArXiv:1707.06347 2017.608
116. Mnih, V.et al.. Human-level control through deep reinforcement learning. Nature 2015,518, 529–533.609
117.
Gu, S.; Lillicrap, T.; Sutskever, U.; Levine, S. Continuous deep Q-learning with model-based acceleration.
610
33rd International Conference on Machine Learning, ICML 2016,6, 4135–4148.611
118.
Rabault, J.; Ren, F.; Zhang, W.; Tang, H.; Xu, H. Deep reinforcement learning in fluid mechanics: A
612
promising method for both active flow control and shape optimization. Journal of Hydrodynamics
2020
,
613
32, 234–246.614
119.
Garnier, P.; Viquerat, J.; Rabault, J.; Larcher, A.; Kuhnle, A.; Hachem, E. A review on deep reinforcement
615
learning for fluid mechanics. Computers & Fluids 2021,225, 104973.616
120.
Bucci, M.A.; Semeraro, O.; Allauzen, A.; Wisniewski, G.; Cordier, L.; Mathelin, L. Control of chaotic
617
systems by deep reinforcement learning. Proceedings of the Royal Society A 2019,475, 20190351.618
121.
Paris, R.; Beneddine, S.; Dandois, J. Robust flow control and optimal sensor placement using deep
619
reinforcement learning. Journal of Fluid Mechanics 2021,913.620
122.
Beintema, G.; Corbetta, A.; Biferale, L.; Toschi, F. Controlling Rayleigh–Bénard convection via reinforcement
621
learning. Journal of Turbulence 2020,21, 585–605.622
123.
Tang, H.; Rabault, J.; Kuhnle, A.; Wang, Y.; Wang, T. Robust active flow control over a range of Reynolds
623
numbers using an artificial neural network trained through deep reinforcement learning. Physics of Fluids
624
2020,32, 053605, [https://doi.org/10.1063/5.0006492]. doi:10.1063/5.0006492.625
124.
Xu, H.; Zhang, W.; Deng, J.; Rabault, J. Active flow control with rotating cylinders by an artificial neural
626
network trained by deep reinforcement learning. Journal of Hydrodynamics 2020,32, 254–258.627
125.
Fan, D.; Yang, L.; Wang, Z.; Triantafyllou, M.S.; Karniadakis, G.E. Reinforcement learning for bluff body
628
active flow control in experiments and simulations. Proceedings of the National Academy of Sciences
2020
,
629
117, 26091–26098, [https://www.pnas.org/content/117/42/26091.full.pdf]. doi:10.1073/pnas.2004939117.
630
126.
Ren, F.; Rabault, J.; Tang, H. Applying deep reinforcement learning to active flow control in weakly
631
turbulent conditions. Physics of Fluids 2021,33, 037121.632
127.
Rabault, J.; Kuhnle, A. Accelerating deep reinforcement learning strategies of flow control through a
633
multi-environment approach. Phys. Fluids 2019,31, 094105.634
Version January 4, 2022 submitted to Fluids 16 of 16
128.
Belus, V.; Rabault, J.; Viquerat, J.; Che, Z.; Hachem, E.; Reglade, U. Exploiting locality and translational
635
invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling
636
liquid film. AIP Advances 2019,9, 125014, [https://doi.org/10.1063/1.5132378]. doi:10.1063/1.5132378.637
©
2022 by the authors. Submitted to Fluids for possible open access publication under the terms and conditions
638
of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).639
