The Lattice Boltzmann Equation Method for Complex Flows

Abstract

The lattice Boltzmann equation (LBE) method is a promising technique for simulating fluid flows and modeling complex physics. Because the LBE model is based on microscopic models and mesoscopic kinetic equations, it offers many advantages for the study of multi-component or multiphase flows. However, there are still challenges encountered when dealing with thermal effects and multiphase flows, particularly at small scales or in varying geometries. In this paper, we discuss some techniques to overcome these challenges. First, we present an overview of the LBE method, and show how it can be extended to model multiple phases and thermal effects. Next, we describe our multi-component and multiphase (MCMP) LBE method for high density ratios. While the original formulation of Shan and Chen’s (SC) model can incorporate some multiphase and component scenarios, the density ratio of the different components is restricted (less than approximately 2.0), which limits the applications. Hence, based on the SC model and improvements in the single-component multiphase (SCMP) flow model reported by Yuan and Schaefer, we have developed a new model that can simulate a MCMP system with a high density ratio. An example of that system is shown. Finally, we have developed a parallel computation LBE method based on the Compute Unified Device Architecture for NVIDIA GPUs. Using this method, we are able to efficiently model a number of phases and length scales, examples of which are presented.

Full text

1 Copyright © 2012 by ASME
Proceedings of the ASME 2012 10th International Conference on Nanochannels, Microchannels and Minichannels
ICNMM2012
July 8-12, 2012, Rio Grande, Puerto Rico
ICNMM2012-73049
THE LATTICE BOLTZMANN EQUATION METHOD FOR COMPLEX FLOWS
Laura Schaefer
University of Pittsburgh
Pittsburgh, PA, USA
Michael Ikeda
University of Pittsburgh
Pittsburgh, PA, USA
Jie Bao
Pacific Northwest National Lab
Richland, WA, USA
ABSTRACT
The lattice Boltzmann equation (LBE) method is a
promising technique for simulating fluid flows and modeling
complex physics. Because the LBE model is based on
microscopic models and mesoscopic kinetic equations, it offers
many advantages for the study of multi-component or
multiphase flows. However, there are still challenges
encountered when dealing with thermal effects and multiphase
flows, particularly at small scales or in varying geometries. In
this paper, we discuss some techniques to overcome these
challenges. First, we present an overview of the LBE method,
and show how it can be extended to model multiple phases and
thermal effects. Next, we describe our multi-component and
multiphase (MCMP) LBE method for high density ratios. While
the original formulation of Shan and Chen’s (SC) model can
incorporate some multiphase and component scenarios, the
density ratio of the different components is restricted (less than
approximately 2.0), which limits the applications. Hence, based
on the SC model and improvements in the single-component
multiphase (SCMP) flow model reported by Yuan and Schaefer,
we have developed a new model that can simulate a MCMP
system with a high density ratio. An example of that system is
shown. Finally, we have developed a parallel computation LBE
method based on the Compute Unified Device Architecture for
NVIDIA GPUs. Using this method, we are able to efficiently
model a number of phases and length scales, examples of which
are presented.
Keywords: lattice Boltzmann, multiphase, thermal effects
1.0 INTRODUCTION
Thermal fluid flows with multiple phases and multiple
components are ubiquitous, but the dynamics of these systems
are not well understood. Thus, the study of thermal flows is of
great interest from the perspective of both fundamental
scientific research and engineering applications. The field of
computational fluid dynamics (CFD) concerns the numerical
study of transport phenomena through the solution of nonlinear
partial differential equations (nPDEs) in discrete space and
time. The equations to be solved, and the manner in which they
are discretized, can vary greatly depending on the problem of
interest. Over the last 50 years, the solution of the Navier-
Stokes (NS) equations using finite volume and finite difference
approaches has dominated the fluid flow modeling community.
However, in the past two decades, alternative approaches have
arisen and are beginning to take hold, due to their ability to
capture more complex flow dynamics.
While traditional fluid dynamics and thermodynamics
give descriptions of the macroscopic transport in a system, they
do nothing to describe the molecular behaviors that cause the
macroscopic effects that are seen. Kinetic theory, on the other
hand, uses the knowledge that fluids are composed of a large
number of molecules to describe why flows behave the way
they do [1]. Each molecule is treated as a particle with a
position and a velocity. All of the particles are then allowed to
flow through the domain and collide with other particles,
obeying classical mechanics. This process of streaming and
collision, when averaged over the very large number of
molecules in a system, gives a fluid its macroscopic properties
of density, velocity, and temperature.
The most exact manner in which fluid flows can be
numerically simulated is the Molecular Dynamics (MD)
approach. As the name implies, this method uses the actual
dynamics of individual molecules, allowing each one to move
and collide according to Newton’s equations of motion.
Potential functions are used to describe the interactions between

2 Copyright © 2012 by ASME
molecules, and the many-body problem is solved through time,
deterministically. However, realistic systems are comprised of a
huge number of molecules and these simulations quickly
become massive in size and computational difficulty.
Consequently, only very small systems, with limited numbers of
molecules, are capable of being modeled with current
computational resources. As computing power increases, these
simulations will become more powerful as they take into
account the real system dynamics from a molecular perspective.
At this time though, the MD approach is simply not feasible for
macroscale flow simulations. Furthermore, from the perspective
of macroscale dynamics, the great detail captured in MD
simulations is, to a large extent, unnecessary.
To reduce the complexity of the MD models, mesoscopic
approaches are being developed. A mesoscopic model describes
behaviors in between the macroscopic and the microscopic.
Instead of completely neglecting molecular interactions or
meticulously accounting for the motion of each molecule, a
mesoscopic model deals with the averages of molecular
behaviors. This concept is fundamental to the field of statistical
mechanics. Pioneered by Ludwig Boltzmann in the second half
of the 17th century, statistical mechanics claims that given the
extremely large number of particles in a macroscopic system,
not only is a statistical approach necessary, but it is also more
appropriate. The information lost through these assumptions is
negligible in regard to the overall system dynamics and the
reduction in computational difficulty is immense.
The lattice Boltzmann equation (LBE) method was
developed from this perspective. Starting from the MD
approach described above, the individual molecules can be
lumped into groups and only allowed to move in predefined
directions. Each group can then be represented by functions of
the probability that it exists at a certain location and time, with a
particular velocity. The evolution of those functions through
space and time is described mathematically by the Boltzmann
transport equation. The power of the LBE method to solve
complex flows is only beginning to be recognized. Its potential
is evidenced by the interest forming in industry for simulations
based on these models. In fact, Porsche AG recently formed a
partnership with Exa Corporation, one of the sole providers of
commercial LBE CFD code, for use in their product
development process [2].
In this paper, we will first present an overview of the basis
of the LBE method, and then discuss extensions to the basic
model, so that multi-phase and thermal effects can also be
simulated. Next, we will present our extension of the LBE
method to multi-phase and multi-component flows. A simple
simulation of a droplet in a vapor field will be shown, in order
to demonstrate the advantages of our method. Finally, we will
show how the LBE method (in all of its permutations) benefits
from parallelization over multiple graphical processor units
(GPUs). Two examples will be presented that show a 13-20
time decrease in solution time.
2.0 NOMENCLATURE
F force
G interaction intensity
P pressure
T temperature
a acceleration, EOS coefficient
b EOS coefficient
c lattice speed (without subscript), weighting coefficient
(with subscript)
e discrete velocity set
f particle distribution function
g interparticle force
t time
u macroscopic velocity
w weighting factor
x position

EOS temperature function, thermal diffusivity
β volumetric thermal expansion coefficient

surface tension strength

relaxation time

viscosity

macroscopic density

relaxation time
Ω collision integral

acentric factor
ξ discrete velocity
ψ effective mass
Subscripts

location
b body
c critical
s surface tension
w wall
Superscripts
eq equilibrium
T temperature
3.0 THE LATTICE BOLTZMANN EQUATION METHOD
The LBE model is derived from the continuum Boltzmann
equation, which is an integro-differential equation, and
describes the evolution of a single-particle distribution function
(PDF) in the physical-momentum space:


ff
t
fx


(1)
The particle distribution function (f) is a function of
position (x), velocity (ξ), and time (t). The Boltzmann equation
also incorporates the acceleration (a) and a collision integral
(Ω). The collision integral describes the manner in which the
collisions between particles modifies their PDFs.

3 Copyright © 2012 by ASME
Because of the high dimensions of the distribution and the
complexity in the collision integral, direct solution of the full
Boltzmann equation is a formidable task for both analytical and
numerical techniques [3]. In 1954, Bhatnagar, Gross and Krook
developed the Boltzmann-BGK equation which is an important
simplification of the original Boltzmann equation [4]. The
Boltzmann-BGK equation takes the form:
 
0
1fff
t
f




(2)
For solving
f
numerically, the Boltzmann-BGK equation
is first discretized in the momentum space using a finite set of
velocities



:
 
)(
1eq
fff
t
f







(3)
where
   
txftxf ,,,
 
 
and
   
txftxf eq ,,, )0()(
 
 
are the
distribution function and the equilibrium distribution function
of the

th discrete velocity


, respectively. The equilibrium
distribution function can be expressed in the form:
 





 uu
c
ue
c
ue
c
wf eq 
2
2
42 23
293
1
 
(4)
where

w
is the weighting factor,
t
x
c



is the lattice speed,

e

is the discrete velocity set, and
u

and

are the
macroscopic velocity and density [5].
The weighting factors and discrete velocities for the
D2Q9 model (shown in Fig. 1) are, for example:
 
   
 









.8,7,6,5,1,1
;4,3,2,1,1,0,0,1
;0,0,0




c
cce

(5)









.8,7,6,5,36/1
;4,3,2,1,9/1
;0,9/4




w
(6)
After discretizing the PDF in momentum space, the number of
possible particle spatial positions and microscopic momenta are
reduced to nine for the 2-D problem.
For 3-D flow, there are several cubic lattice models, such
as the D3Q15, D3Q19 and D3Q27 model, as shown in Fig. 2.
The weighting factors and discrete velocities for D3Q19 (a
widely used state space) are:
 
     
     









.18,...,8,7,1,1,0,10,1,0,1,1
;6,...,2,11,0,0,0,1,0,0,0,1
;0,0,0,0




ccc
ccce

(7)









.18,...,8,7,36/1
;6,...,2,1,18/1
;0,3/1




w
(8)
Based on the D2Q9, D3Q15, D3Q19 and D3Q27 frame
definition, the density and the velocity can be defined as:


bf
0


(9)


befu 0
1



(10)
where b represents the total number of possible particle spatial
positions. To solve
 
txf ,


, Eq. (3) needs to be further
discretized in physical space
x

and time t, so the completely
discretized form of Boltzmann-BGK equation is:
       
 
txftxftxftttexf eq ,,
1
,, 
 

(11)
where

is the non-dimensional relaxation time. This equation
often can be solved using the following two steps [6]:
Figure 1. The 2-D grids for the LBE method
Figure 2. 2-D square lattice and 3-D cubic lattice for LBM

4 Copyright © 2012 by ASME
Collision:
       
 
txftxftxftxf eq ,,
1
,,
~
 

(12)
Streaming:
   
txftttexf ,
~
,


(13)
After the streaming step (Eq. (13)), we can substitute the
newly calculated PDF
 
tttexf  ,


into Eq. (12) by
replacing the old
 
txf ,


. Every loop is a time step, and after
many time steps (which may be over 10,000 for particularly
complicated phenomena), the program will converge.
4.0 THE MULTI-PHASE LBE METHOD
It is commonly accepted that the separation of different
phases or components is microscopically due to the long-range
interaction between the molecules of a fluid [7]. This interaction
can be expressed as:
   
xgxcxF 


 0
)(
(14)
where c0 is a constant depending on the lattice structure. For the
D2Q9 and D3Q19 lattices, c0 = 6.0, and for the D3Q15 lattice,
c0 = 10.0. The coefficient for the strength of the interparticle
force is g, with g < 0 representing an attractive force between
particles and g > 0 a repulsive force.
 
x


is the effective
mass, which is a function of local density and can be varied to
reflect different fluid and fluid mixture behaviors, as
represented by various equations of state (EOS). This equation
is derived from the original Shan and Chen (SC) model [8].
Although that work only used the interparticle forces of nearest
neighbor sites, it can be extended to include other neighboring
sites as long as the gradient term


is properly specified. We
use both the nearest and next-nearest sites to evaluate this
gradient term, which gives a six-point scheme for two
dimensions:
     
 
 

     

1,11,11,1
1,1[,1,1
,21




jijiji
jicjijic
xji



(15a)
     
 
 
     
]1,11,11,1
1,1[1,1,
,21




jijiji
jicjijic
yji



(15b)
where
1
c
and
2
c
are the weighting coefficients for the nearest
and next nearest sites, respectively.
In addition to the interparticle forces, if the problem
includes a solid wall boundary, the interaction between the fluid
and solid interface needs to be considered, so the forces applied
on a particle that contacts the solid wall are:
        
xxxxxGxxF w
xww



 


,
(16)
where
 
xxGw
 ,
reflects the intensity of the fluid-solid
interaction, and
 
x
w


is the wall density, which equals one at
the wall and zero in the fluid. Furthermore, in addition to
interparticle and wall forces, the body force can be defined as:
   
axxFb




(17)
The viscosity and the surface tension are two additional
important factors for specifying fluid characteristics. The
viscosity is defined in the LBE model as:
tcs

2
2
1





(18)
where
s
c
is the speed of sound in the LBE model. Hence, the
viscosity can be changed by choosing a different relaxation time

. In order to adjust the surface tension, an additional force
term should be introduced into the fluid-fluid interaction, and is
defined as:
   

2

s
F

(19)
where

determines the strength of the surface tension [9].
Hence, the total force on each particle can be expressed
as:
 sbwitotal FFFFF 
(20)
All of these forces can be incorporated into the model by
shifting the velocity in the equilibrium distribution. This means
that the velocity
u

in Eq. (4) is replaced by:
 
x
F
uu total
eq 





(21)
The effective mass

, which was mentioned at the
beginning of this section, is the mechanism for incorporating a
more sophisticated EOS. As stated previously, the effective
mass
))(()( xx 


is a function of the local density, and can
be defined as:
 
gc cp s
0
2
2
)(




(22)
where p is the pressure. The choice of EOS can reflect the
relationship between the pressure, temperature and density. In
Yuan and Schaefer’s work [10], five different EOS were tested
in this model, and it was found that that Peng-Robinson (P-R)
EOS provided the maximum increase in the density ratio of
Single Component, Multi-Phase (SCMP) flows while

5 Copyright © 2012 by ASME
maintaining small spurious currents around the interface. The P-
R EOS is expressed as:
22
2
21 )(
1




bb Ta
b
RT
P



(23)
 
 
 
2
2/126992.05422.137464.01)( c
TTT 

(24)
with
c
c
PTR
a22
45724.0

and
c
c
PRT
b0778.0

, where
a
is the
attraction parameter, b is the volumetric or repulsion parameter,
and

is the acentric factor. TC and PC are the critical
temperature and critical pressure, respectively.
5.0 THE THERMAL LBE METHOD
It can be seen from Eq. (23) that the effective mass can be
a function of temperature, as well as pressure and density. Also,
as stated previously, the LBE method can be used to solve
generalized NPDEs. Hence, the LBE model can also be applied
to solving thermal problems. In a single-phase thermal fluid
system, if the viscous and compressive heating effects are
negligible, the temperature field satisfies a much simplified
passive-scalar equation:
 



TTu
t
T
(25)
where
u

is the macroscale velocity,

is the thermal
diffusivity, and

is the source term. To solve Eq. (25) using
the LBE method, we must first define a particle distribution
function (PDF)
 
txf T,


for temperature, which is the same as
the dynamic PDF. The temperature then can be found using the
following relationship:


bT
fT 0

(26)
Through Eqs. (27) and (28) below, we can solve the
temperature PDF
 
txf T,


numerically:
     
 
 
 
txftxftxftttexf eqTT
T
TT ,,
1
,, 
 

(27)
 
 





 uu
c
ue
c
ue
c
TwfeqT
2
2
42 23
293
1

(28)
where
T

is the dimensionless single relaxation time for
temperature. The temperature variance results in a buoyancy
force, which is expressed as:
 
jTTgG 

0


(29)
To incorporate the buoyancy force, we substitute Eq. (29)
into Eq. (20), calculate the total force that is applied on the
particles, and then calculate the shifted velocity by Eq. (21).
Finally, the velocity
u

in Eq. (28) is replaced by this shifted
velocity. Through including the buoyancy force, the LBE
method can be used for a number of systems with thermal
effects, such as natural convection problems.
6.0 MULTI-COMPONENT MULTI-PHASE (MCMP) LBE
METHOD WITH HIGH DENSITY RATIOS
6.1 Explanation of the Method
Although the previous sections have focused on single
component flows, because the LBE model is based on
microscopic models and mesoscopic kinetic equations, it offers
many advantages for the study of multi-component as well as
multi-phase flow problems. While the original formulation of
Shan and Chen’s (SC) model can simulate some multi-phase or
multi-component examples, the density ratio of the different
components is greatly restricted, to less than approximately 2.0.
This obviously limits the applications of this multi-component
multi-phase (MCMP) LBE model. Hence, based on the original
SC MCMP model and the improvements in the single-
component multi-phase (SCMP) flow model reported by Yuan
and Schaefer [10], we have developed a new model that can
simulate a MCMP system with a high density ratio.
As in the single component multi-phase flow case, the
separation of different components in MCMP flows is also due
to the long-range interaction between the molecules of the fluid
[11], so the interaction force must be revised to also include two
parts for a multi-component fluid. One contribution is the
interaction between molecules of the same component, and
another is the interaction between molecules from different
components. In a similar manner to Eq. (14), these two parts
can be expressed as:
   
xgxcxF iiiiii



 0, )(
(30a)
   
xgxcxF jijiji



 0, )(
ji 
(30b)
where
F
i,i
is the force between the different particles of
component i, and
F
i,j
indicates the force between the
component i and component j.
To increase the maximum possible density ratio between
the different components, one first should increase the density
ratio for the different phases of each single component. Work
has already been done on increasing the density ratio for SCMP
flows. For example, as reported by Swift [12], the maximum
density ratio obtained using the free-energy-based approach is
less than 10:1, and the largest density ratio tested in the He,
Chen and Zhang (HCZ) approach is 40:1 [13]. These are
improvements, but are still not large enough for most practical
problems. Yuan and Schaefer [10] found, however, that is
possible to simulate SCMP flows with a density ratio that can
reach 1000:1 by using a more accurate EOS, such as the van der

6 Copyright © 2012 by ASME
Waals, Peng-Robinson, or Carnahan-Starling EOS [14], and all
of which can be easily applied to the LBE model. Achieving a
density ratio of up to 1000:1 means that the LBE model can be
used for the simulation of most single-component vapor-liquid
flows.
Unlike in the original SC model, though, the coefficient of
interaction strength within a component (gii) here cannot control
the overall interaction strength. (Indeed, it is canceled out when
Eq. (22) is substituted into Eq. (30a).) The only strict
requirement for gii is to ensure that the whole term inside the
square root in Eq. (22) is positive [15]. However, we have
found that the coefficient of interaction strength between
different components gij is very important for creating and
extending the MCMP LBE model. Firstly, when Eq. (22) is
substituted into Eq. (30b), gij is not eliminated. Secondly, from
Eq. (30b), it can be seen that gij affects the magnitude of the
interparticle force between different components Fi,j. The
behavior of the interaction between the different components is
primarily controlled by this force, so interaction can be adjusted
through changing the value of gij. From our tests, this force
plays a critical role in adjusting the system density ratio. More
details of these tests can be found in [16], but one case will be
presented here for illustration purposes.
6.2 A Droplet in Equilibrium, Without Body or External
Forces
The first example is the simulation of a circular droplet in
a 100 x 100 2-D square domain for a liquid-gas system without
body forces. A periodic boundary condition is applied on the
two vertical boundaries (x = 0 and x = 100), and a bounce back
treatment is used for the upper and bottom boundaries (y = 0
and y = 100). For an initial condition for component 1 (the
liquid component), a low density value (

= 0.005) was
specified for most of the simulation region, except for a circular
area at the center of the simulation space, where a high density
value was set (

= 1). For component 2 (the gas component),
density was set to a relatively higher value than component 1 (

= 0.01) at most areas of the simulation domain, except for the
center circular area (corresponding to the liquid component),
where an extremely low density was used (

= 0.0001).
It should be noted that, generally, if the problem can reach
a converged result, the initial shape of the high density area is
not important. A square, triangle, or any random geometry will
not affect the final result. Because a droplet always tends to
become the shape that has smallest surface area, for this 2-D
case, the droplet will always become a circle. Hence, setting the
initial shape to a circle increases the problem’s convergence
speed, since, as with most numerical methods, the smaller the
difference between the initial conditions and final result, the
faster the model converges.
Figure 3(a) shows the density contour of component 1 (the
liquid component), and Fig. 3(b) shows the comparison of the
density of the two components along the centerline (y = 50, 0 ≤
x ≤ 100) on a log10 scale. For convenience, we have highlighted
several segments on these lines, which are the maximum density
of component 1, minimum density of component 1, maximum
density of component 2, and minimum density of component 2,
noted as ρComponent1,max, ρComponent1,min, ρComponent2,max, and
ρComponent2,min, respectively. The density variation in each
segment is very small compared to the density change at the
interface of the different components, so this small variation can
be neglected. In Fig. 3(b), the density of component 1 in the
droplet (ρComponent1,max) is on the order of 101, while the density
of component 2 around the droplet (ρComponent2,max) is on the
order of 10-2. Hence the density ratio of these two components
is around 1000. In general, the density ratio of a system refers
to the ratio between the maximum densities of the two
components (
max,2
max,1
Component
Component


).
As detailed in Section 6.1, the balance of the weights of
the two parts
F
i,i
and
F
i,j
directly affects the density ratio of the
different components. The forces
F
i,i
and
F
i,j
vary in the
simulation domain and reach a maximum value around the
interface of the two components. Figure 4 shows the
(a)
(b)
Figure 3. (a) Density contour for a circular droplet, and
(b) Comparison of the densities of the two components
along the center line (y = 50, 0 ≤ x ≤ 100)

7 Copyright © 2012 by ASME
distribution of the ratio of
x
F1,1
and
max,1,1
F
in the test area,
where
x
F1,1
is the x-direction of the interparticle force of
component 1 and
F
1,1,max
is the maximum force in the entire test
region. Hence, we find that the ratio of the maximum forces
(
F
1,1max
F
1,2max
) is a key factor, and examine how this ratio affects the
density ratio of the two components.
Figure 5 shows the relationship between the density ratio
and maximum force ratio. From Fig. 5, we can see that by
changing the ratio of the maximum force, the density ratio can
be adjusted from 1 to 1000. This is a substantial range that can
cover many kinds of liquid-gas or liquid-liquid two component
systems. One should notice that the forces
F
1,1
and
F
1,2
are in
conflict when they are of the same order. Therefore, if the
density ratio should be around 1, the interparticle force between
the different components
F
1,2
is 1 or 2 orders of magnitude
higher than the force
F
1,1
(or
F
1,1
can be directly neglected).
This coincides with the actual physical phenomenon.
When the density ratio of two components is very large, such as
in a water-air system, the formation and position of the interface
are mainly controlled by the tension of the water (as well as the
body force and the interaction between the fluid and solid wall,
if those are present). The interparticle force between the
different components is very weak, especially for steady-state
behavior. With a decrease in the density ratio, the interparticle
force becomes more important. When the density ratio is near
1.0, as in a water-oil system, the formation and position of the
interface is mainly caused by the interparticle force between the
different components instead of the tension, body, force or other
forces applied on each single component.
7.0 PARALLELIZING THE LBE METHOD USING THE
COMPUTE UNITED DEVICE ARCHITECTURE (CUDA)
7.1 Motivation
Although the development of new computer techniques
has increased the efficiency of numerical simulation in recent
years, the time cost of many computational problems is still
unaffordable. As discussed in the introduction, the LBE method
is less time consuming than an MD simulation, but it can still
consume a prohibitive amount of processor time as the
simulation conditions become more complex. This time
consumption problem can become particularly restrictive for
simulations based on the 3-D, multi-component, and thermal
lattice Boltzmann equation (LBE) model. One obvious way to
reduce the calculation time is through parallel computation.
However, recent research [17] has shown that application of the
multi-thread LBE model on a traditional multi-core or multi-
CPU computer system does not lead to a large increase in
solution speed. This is because the LBE model requires a
greater memory bandwidth than traditional CFD methods, and a
traditional multi-core or multi-CPU computer system cannot
offer enough memory bandwidth for the 3-D LBE model. A
system based on the Intel Core 2 CPU [18], which is a popular
CPU for personal computers and mainstream workstations, can
only increase the speed by approximately 2.68 times when the
number of cores increases from 1 to 8. Hence, an increase in the
number of cores cannot offer a linear speed increase. Also, an
increase in the number of cores causes the price of the system to
rise exponentially. Although some newly developed multi-core
computer systems, such as the IBM CELL system [19], partially
solve the memory bandwidth problem, its impressive
computational efficiency comes with a high price and a difficult
programming environment that is a major departure from
conventional programming.
For the purpose of increasing simulation speed without a
correspondingly large increase in cost, we have developed a
multi-thread parallel lattice Boltzmann equation model based on
the Compute Unified Device Architecture (CUDA). CUDA was
developed by NVIDIA, and is a hardware and software
architecture for issuing and managing computations on a
graphic processor unit (GPU) as a data-parallel computing
device [20]. With multiple cores and high memory bandwidth,
today’s GPUs can be used for both graphics and non-graphics
applications, and have many advantages over traditional CPUs.
Figure 4. Distribution of the ratio of
x
F1,1
and
max,1,1
F
Figure 5. Density ratio
2
1
Compo nent
Compo nent


variation with respect to the
ratio of the maximum force
max2,1
max1,1
F
F

8 Copyright © 2012 by ASME
This is because the structure of a GPU is especially well-suited
to address problems that can be expressed as data-parallel
computations (the same program is executed on many data
elements in parallel) with high arithmetic intensity (the ratio of
arithmetic operation to memory operations).
With the development of the GPU and the optimization of
software for that programming environment in the last two
years, CUDA has become a very powerful and popular parallel
computational solution for various kinds of practical
applications. As stated above, the memory bandwidth has
limited the multi-thread LBE model’s speed in the traditional
multi-CPU system. Most desktop systems and workstations can
only offer less than 20 GB/s of memory bandwidth, while the
GPU system can easily offer more than 50 GB/s memory of
bandwidth, and one current high-end graphic card, NVIDIA
GEFORCE GTX280, has 141.7 GB/s of memory bandwidth.
7.2 CUDA Programming Strategy
Based on the design and characteristics of GPUs, the
device memory is of much higher latency and lower bandwidth
than on-chip memory, so device memory accesses should be
minimized when implementing the LBE model. A typical
programming pattern is shown below:
1. Load data from device memory to shared memory,
2. Process the data in shared memory, and
3. Write the results back to device memory.
The main idea for the parallel computational LBE method is
launching as many as possible threads at the same time for
updating the variables on each lattice. Figure 6 demonstrates the
parallel strategy of our program. In one calculation cycle, each
thread only calculates the data on one grid, and each thread
reads the data to shared memory for the calculation of one
certain grid only. This method can also avoid encountering the
block interface problem.
In order to demonstrate this clearly, the number of threads
is set to 10 in the example. In the first step, the 10 threads
update the value on the grid of blocks 1 to 10 (they are in grey),
as shown in Fig. 6(a), and in the second step, the 10 threads
update the value on the grid of blocks 11-20 (shown in grey), as
seen in Fig. 6(b). By continuing this method (Figs. 6(c) and
6(d)), all of the data in the simulation domain can be updated.
Through this method, it is easier to deal with the division of the
simulation domain, especially when using a large number of
threads. The workload of each thread can be more easily
regulated than in the traditional division method shown in Fig.
7. Furthermore, the shape of each block is not required to be
rectangular in our method, so it is more efficient to deliver the
workload to all of the threads that are available on the
hardware.
7.3 Implementation of CUDA for the LBE Method
For the simplest case, single-component single-phase
flow, the simulation can be divided into two steps, a collision
and a streaming step, as shown in Eqs. (12) and (13) in Section
3.0. The collision step is completely local, but the streaming
step (Eq. (13)) is not. Figure 8 illustrates the meaning of the
streaming step. It can be seen as a simple copy step. Each
component of the PDF (
f1,f2,..., f8
) is copied to the nearby
grids after a streaming step. For the CUDA program, all of the
data must be copied from the device memory to shared memory
for processing, so the streaming step can be done during this
(a)
(b)
(c)
(d)
Figure 6. Demonstration of block division for CUDA
Figure 7. Division of blocks
(a) (b)
Figure 8. Streaming step: (a) time step 0; (b) time step 1

9 Copyright © 2012 by ASME
copying procedure. Figure 9 shows the process for the
streaming step and collision step.
A similar division of calculation steps for the thermal and
multi-component multi-phase models can be seen in Figs. 10
and 11. The main complication in these schemes is the proper
calculation of the gradient of the effective mass (

ψ). We use
both the nearest and next-nearest sites to evaluate this gradient
term, which gives a six-point scheme for two dimensions, as
previously outlined in Eqs. (15a) and (15b). This gradient term
depends on the data of the six nearby grids. Hence, we must
copy all of the six points’ data from device memory to shared
memory for the calculation of this gradient term. If the gradient
term in the grey grids (using 10 threads as an example) is
required to be calculated, as shown in Fig. 12, the values of the
effective mass in the grids circled by the bold black line need to
be copied from device memory to shared memory. After
calculating this gradient term, the value of

ψ can be copied
back to device memory or restored in shared memory
temporarily for calculating the interparticle forces. When
calculating interparticle forces on each grid (i,j,k) using Eq.
(14), each thread reads the effective mass
kji ,,

and gradient of
effective mass
kji ,,


that are stored in device memory. Hence
this step is highly parallel. In the following two sections, we
will demonstrate the resulting improved performance for both
single-phase and multi-phase flows from the parallelization of
the LBE method over multiple GPUs.
7.4 Example Case: Single-Phase Flow
Our tests were run on two different NVIDIA GPUs, an 8800
GTS and a 260 GTX. These GPUs represent the middle range
of performance of the two generations of NVIDIA GPUs for
Figure 9. Demonstration of the streaming step and collision step based on the CUDA LBE model
Figure 10. Demonstration of the Thermal LBE model based on CUDA programming

10 Copyright © 2012 by ASME
CUDA, and were the most popular GPUs due to their market
price. The specifications for each of these GPUs are found in
Table 1.
Table 1. GPU Specifications
Main Clock
RAM
Mem.
Clock
Mem. Bus
Width
Cores
8800 GTS
1.2GHz
320MB
1.8GHz
256bit
96
260 GTX
1.4GHz
896MB
2.1GHz
448bit
216
For comparison, we also performed test runs on two
different Intel dual-core CPU systems, an E6600 (2.4GHz) and
an E8500 (3.16GHz). We used the Compaq Fortran compiler
6.6b and the Visual Studio 2005 C++ compiler for the CPU
code, and used NVCC release 2.0 for the GPU code.
The first case is an external flow through an area of
porous media, as shown in Fig. 13(a), with 50x50x100 grids. A
periodic boundary condition is used for the boundaries of y=0,
y=ymax, z=0, and z=zmax. A uniform inlet velocity is at x=0, and
an extended boundary treatment is used for x=xmax. Figure 13(b)
shows the velocity profile for a slice of the flow at y=25. Figure
14 shows the calculation time comparison between one of the
GPUs and the CPUs. All of the codes were run for 10,000 time
steps to reach steady state. It can be seen that the simulation on
the NVIDIA 260 GTX took approximately 5% of the time of
the simulation on the fastest CPU.
7.5 Example Case: Multi-Phase Flow
The ability to create detailed simulations for multi-phase
flow is one of the greatest strengths of the LBE method, as
outlined in the previous sections. By applying the parallel
computational method that was introduced in Section 7.3, 3-D
multi-phase problems can be solved. One example of this is the
simulation of the melding process of nanocomposites. Polymer-
based nanocomposites are becoming an attractive set of
materials due to their multiple functions and many potential
applications [21]. These materials are expected to possess
unique electric, magnetic, optical, and mechanical properties
that can be significantly different from those of an individual
material [22]. For polymeric matrix composites, the assembly of
composite materials is the key to success. However, the intrinsic
van der Waals attraction among nanowires and the high surface
area and high aspect ratio of nanowires often prevent efficient
filling of a polymer fully into the matrix, especially for
materials such as ZnO.
The focus of most work for this melding process is how to
insert a polymer liquid into the host nano-framework [23], such
as a nano-wire array [24-25], as shown in Fig. 15(a), as fully or
as deeply as possible. Figure 15(b) shows a schematic diagram
of an ideal ZnO nanocomposite. Figure 15(c) shows an SEM
image of ZnO nanocomposites where pure polyimide PI-2611 is
Figure 12. Grids for calculating the gradient term of the
effective mass
(a) (b)
Figure 13. (a) Demonstration of simulation domain, and
(b) Velocity profile at y=25
Figure 14. Calculation time for 3-D porous media flow
Figure 11. Demonstration of MCMP LBE model based on CUDA programming

11 Copyright © 2012 by ASME
used to fill the matrix, and the polyimide almost cannot be
inserted into the nanowire array. Figure 15(d) shows another
SEM image of ZnO nanocomposites where polyimide PI-2611
is diluted by the solvent T9039 to a 1:2 weight ratio, and the
polyimide can partially penetrate into the array. Our LBE model
can show how the liquid flows into the nanowire array, and how
deeply the polymers can be inserted while adjusting the
attractive or repulsive interaction between the solid host and
liquid polymers. Our second case for demonstrating the CUDA
LBE model gives an example of solving this practical problem.
Figure 16 shows the simulation domain, which contains
100x100x50 grids. The small pillars, which represent the
nanowire array, are planted on the bottom wall. The diameter of
each pillar is 3 grids. The periodic boundary condition is used
on the four vertical boundaries, and the gravity force is along
the negative z-direction. A liquid droplet is placed on the top of
the pillars.
When a strong repulsive force is set between the liquid
and the solid surface, the droplet stays primarily on the top of
the pillar array, and it cannot penetrate the array completely, as
shown in Fig. 17(a). The cross section is for the center line of
the droplet, and the small figure at the upper-left corner shows
the interface of the liquid. However, for an attractive force
between the liquid and the solid, the droplet can flow into the
space between the pillars, as shown in Fig. 17(b). The value of
Gw
controls the strength of the attractive force, so that the
droplet can be partially or fully inserted into the pillar array.
We also tested how the distance between pillars affects
this process. The distribution of a real nanowire array is much
more complicated than Fig. 17, as shown in Fig. 15. The
distance between nanowires and the length of the nanowires
vary, and the nanowires are also not perfectly vertical to the
bottom wall. Obviously, all of these variations would affect the
melding process. Figures 18(a) and 18(b) show the results for
different distances between the pillars, and Fig. 18(c) shows a
result for a pillar array with varied lengths and end geometries.
These are just a few examples of how the geometries can
be varied for these arrays. Our numerical model has the
capacity to be applied to many more complex solid boundary
conditions, and therefore offers a useful tool for studying how
Figure 16. Sketch of the simulation domain
(a) (b)
Figure 17. (a) Cross section and liquid interface of the droplet
for repulsive interaction, and (b) cross section and liquid
interface of the droplet for attractive interaction
(a)
(b)
(c)
Figure 18. Cross-section of droplet for different distributions:
(a) the distance is 4 grids, (b) the distance is 8 grids, and
(c) the pillars have different lengths
(a) (b)
(c) (d)
Figure 15. (a) SEM image of nanowire array; (b) schematic
diagram of ideal Zno nanocomposite; and (c), (d) SEM image
of ZnO nanocomposites

12 Copyright © 2012 by ASME
each element (such as the length, the distribution of nanowires,
the force between the polymer and nanowire, etc.) affects the
melding process. Moreover, parallelization using CUDA allows
for much faster solution times, so that a range of conditions can
be simulated. Figure 19 shows the calculation time for 1,000
steps for a CPU and a GPU simulation of the varying-length
nanocomposite. The GPU program is 13 times faster than the
CPU program.
8.0 CONCLUSIONS
This work presents a brief overview of the lattice
Boltzmann equation method. The origins of the method in the
Boltzmann equation are shown, as well as its translation to the
lattice formulation. The basic LBE method is described for 2-D
and 3-D cases, and then extended to include multiple phases for
a single component, through using advanced equations of state
to calculate the effective mass. Next, the technique and
implications of introducing thermal effects is presented, which
includes the creation of a second particle distribution function
for temperature. Finally, the model is extended to include not
only multiple phases, but also multiple components.
While the LBE method is relatively computationally
efficient, as the simulations become more complex (for example
with multiple components with thermal effects), it becomes
advantageous to explore methods for decreasing the
computational time. One example of such a method,
parallelization of the LBE method over multiple graphical
processor units using the Compute United Device Architecture,
is presented. The basic framework of the method is discussed,
and then two example cases are presented, using single-phase
and multi-phase simulations. The advantages of using GPUs
(over dual-core or multi-core CPUs) are clear from the resulting
improvement in computational speed for these geometrically
complex cases.
Of course, this paper is not long enough to describe all
aspects of the derivation and implementation of the LBE
method. Most importantly, a review and discussion of the
various boundary condition (BC) treatments has been omitted,
as that is a multifaceted field in and of itself, and could
constitute an entirely separate paper. The correct choice of
boundary condition is essential to the accuracy and solution
time of the LBE method, as it is for all CFD. A further
discussion of a range of BCs and their implementation in the
LBE method can be found in [26-30]. There are also still many
aspects of the LBE method that can be further refined, and
should be the subject of future research. These include potential
additional increases in computational speed from non-CUDA
GPU parallelization, pairing of higher-order and lower-order
LBE applications, and extension of the range in which thermal
effects can be simulated for MCMP flows.
ACKNOWLEDGMENTS
This material is based upon work supported by the National
Science Foundation under Grant Nos. CBET-0238841 and
CBET-0729905, and was also supported by the NSF's
Integrative Graduate Education and Research Traineeship
(DGE-0504335).
REFERENCES
1. Hill, T. L., 1987, An Introduction to Statistical
Thermodynamics, McGraw-Hill series in advanced
chemistry, Dover Publications.
2. Exa Corporation, 2011, website: http://www.prweb.com/
releases/Exa/Porsche/prweb4110304.htm.
3. Pathria, R. K., 1996, Statistical Mechanics, 2nd ed.,
Butterworth-Heinemann.
4. Bhatnagar, P. L., Gross, E. P., and M. Krook, 1954, ―A
model for collision processes in gases. I. Small amplitude
processes in charged and neutral one-component system,‖
Physical Review, vol. 94, pp. 511-525.
5. He, X., and L.-S. Luo, 1997, ―Theory of the lattice
Boltzmann method: From the Boltzmann equation to the
lattice Boltzmann equation,‖ Physical Review E, vol. 56,
no. 6, pp. 6811-6817.
6. d’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand,
P., and L.-S. Luo, 2002, ―Multi-relaxation-time lattice
Boltzmann models in three dimensions,‖ Philosophical
Transactions of the Royal Society of London A, vol. 360,
pp. 437-451.
7. Shan, X.-W., and G. D. Doolen, 1995, ―Multicomponent
lattice Boltzmann model with interparticle interaction,‖
Journal of Statistical Physics, vol. 81, nos. 1/2, pp. 379-
393.
8. Shan, X., and H. Chen, 1994, ―Simulation of nonideal
gases and liquid-gas phase transitions by the lattice
Boltzmann equation,‖ Physical Review E, vol. 49, no. 4,
pp. 2941-2948.
9. Shi, B., He, N., and Z. Guo, 2009, ―Lattice Boltzmann
Model for High-Order Nonlinear Partial Differential
Equations,‖ Computational Physics, arXiv:0907.1720v1.
10. Yuan, P. and L. Schaefer, 2006, ―Equations of state in a
lattice Boltzmann model,‖ Physics of Fluids, vol 18,
article 042101.
11. Qian, Y. H., Succi, S., and S. A. Orszag, 1995, ―Recent
advances in lattice Boltzmann computing,‖ Annual Review
of Computational Physics, vol. 3, pp. 195-242.
12. Swift, M., Orlandini, S., Osborn, W., and J. Yeomans,
1996, ―Lattice Boltzmann simulations of liquid-gas and
Figure 19: Calculation time comparison for 3-D multi-phase
flow for a nanocomposite

13 Copyright © 2012 by ASME
binary-fluid systems,‖ Physical Review E, vol. 54, pp.
5041-5052.
13. Zhang, R., 2000, Lattice Boltzmann Approach for
Immiscible Multiphase Flow, Ph.D. thesis, Department of
Mechanical Engineering, University of Delaware.
14. Wei, Y. S., and R. J. Sadus, 2000, ―Equation of state of
the calculation of fluid-phase equilibria,‖ AIChE Journal,
vol. 46, no.1, pp. 169-196.
15. Deng, B., Shi, B. C., and G. C. Wang, 2005, ―A new
lattice Bhatnagar–Gross–Krook model for convection-
diffusion equation with a source term,‖ Chinese Physics
Letters, vol. 22, no. 2, pp. 267–70.
16. Bao, J., 2009, High Density Ratio Multi-component
Component Lattice Boltzmann Flow Model for Fluid
Dynamics and CUDA Parallel Computation, Ph.D. thesis,
Department of Mechanical Engineering and Materials
Science, University of Pittsburgh.
17. Williams, S., Carter, J., Oliker, L., Shalf, J., and K.
Yelick, 2008, ―Lattice Boltzmann simulation optimization
on leading multicore platforms,‖ International Parallel &
Distributed Processing Symposium, Miami, Florida, USA.
18. Intel Core 2 Duo Processors, 2010, http://www.intel.com/
products/processor/core2duo/index.htm?iid=prod+prod_
core2duo.
19. The Cell project at IBM Research, 2010,
http://www.research.ibm.com/cell/.
20. NVIDIA, 2008, CUDA Compute Unified Device
Architecture Programming Guide.
21. Sanchez, C., Julian, B., Belleville, P., and M. Popall,
2005, ―Applications of hybrid organic-inorganic
nanocomposites,‖ Journal of Materials Chemistry, vol. 15,
pp. 3559-3592.
22. Cheng, H., 2009, Fabrication of Zinc Oxide Micro-
Nanostructures and Their Applications in Gas Sensing
and Nanocomposites, Ph. D. thesis, Department of
Mechanical Engineering and Materials Science, University
of Pittsburgh, 2009
23. Wachhold, M., Rangan, K. K., Billinge, S. J. L., Petkov,
V., Heising, J., and M. G. Kanatzidis, 2000,
―Mesostructured non-oxidic solids with adjustable worm-
hole shaped pores: M-Ge-Q (Q = S, Se) Frameworks
Based on the Tetrahedral [Ge4Q10]4- Clusters,‖ Advanced
Materials, vol. 12, no. 2, pp. 85-91.
24. Parthangal, P. M., Cavicchi, R. E., and M. R. Zachariah,
2006, ―A universal approach to electrically connecting
nanowire arrays using nanoparticles—application to a
novel gas sensor architecture,‖ Nanotechnology, vol. 17,
pp. 3786-3790.
25. Tak, Y., and K. Yong, 2008, ―A novel heterostructure of
Co3O4/ZnO nanowire array fabricated by photochemical
coating method,‖ Journal of Physical Chemistry C, vol.
112, no. 1, pp. 74-79.
26. Bao, J., Yuan, P., and L. Schaefer, 2008, ―A mass
conserving boundary condition for the lattice Boltzmann
equation method,‖ Journal of Computational Physics, vol.
227, no. 18, pp. 8472-8487.
27. Inamuro, T., Yoshino, M., and F. Ogino, 1995, "A non-
slip boundary condition for lattice Boltzmann
simulations," Physics of Fluids, vol. 7, no. 12, pp. 2928–
2930.
28. Zou, Q., and X. He, 1997, ―On pressure and velocity
boundary conditions for the lattice Boltzmann BGK
model,‖ Physics of Fluids, vol. 9, no. 6, pp. 1591–1598.
29. Latt, J., Chopard, B., Malaspinas, O., Deville, M., and A.
Michler, 2008, ―Straight velocity boundaries in the lattice
Boltzmann method,‖ Physical Review E, vol. 77, no. 5,
056703.
30. Latt, J., and O. Malaspinas, 2008, Palabos Project
Documentation: Boundary Conditions, website:
http://www.palabos.org/wiki/.