Efficient model of tumor progression simulated in multi-GPU environment

Abstract

The application of computer simulation as a tool in predicting cancer dynamics (e.g., during anti-cancer therapy) requires tumor models, which are non-trivial and, simultaneously, not computationally demanding. To this end, both the level of details and computational efficiency of the model should be well balanced. The restrictions in computational time are forced by very demanding data adaptation process in the phase of parameters learning and their correction on the basis of incoming medical data. Herein we present very efficient multi-GPU/CUDA implementation of 3D cancer model which allows for simulating both the growth and treatment phases of tumor dynamics. We demonstrate that the interaction between the tissue and the discrete network of blood vessels is a crucial component, which influences considerably the simulation time. Here we present a new solution, which eliminates this flaw. We show also that the efficiency of our model does not depend on the complexity of tumor setup. As an example, we confront the growth of tumor in a simple and homogeneous environment with melanoma evolution, which proliferates in a complex environment of human skin. Consequently, the 3D simulation of a tumor growth to the size of a centimeter requires an hour of computations on a mid-range multi-GPU server.

Full text

Efficient model of tumor progression
simulated in multi-GPU environment
Journal Title
XX(X):1–13
c
The Author(s) 2016
Reprints and permission:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/ToBeAssigned
www.sagepub.com/
Witold Dzwinel, Adrian Kłusek, Marcin Ło´
s and Maciej Paszy´
nski
Abstract
The application of computer simulation as a tool in predicting cancer dynamics (e.g., during anti-cancer therapy)
requires tumor models, which are non-trivial and, simultaneously, not computationally demanding. To this end, both
the level of details and computational efficiency of the model should be well balanced. The restrictions in computational
time are forced by very demanding data adaptation process in the phase of parameters learning and their correction
on the basis of incoming medical data. Herein we present very efficient multi-GPU/CUDA implementation of 3D cancer
model which allows for simulating both the growth and treatment phases of tumor dynamics. We demonstrate that
the interaction between the tissue and the discrete network of blood vessels is a crucial component, which influences
considerably the simulation time. Here we present a new solution, which eliminates this flaw. We show also that the
efficiency of our model does not depend on the complexity of tumor setup. As an example, we confront the growth of
tumor in a simple and homogeneous environment with melanoma evolution, which proliferates in a complex environment
of human skin. Consequently, the 3D simulation of a tumor growth to the size of a centimeter requires an hour of
computations on a mid-range multi-GPU server.
Keywords
tumor modeling, continuous/discrete cancer model, multi-GPU/CUDA implementation, melanoma model, tumor therapy
model
Introduction
Advanced mathematical and numerical models, which
require greater computational power and are implemented in
GPU environment, are consequently being used in clinical
practice Pratx and Xing (2011). Particularly, in oncology,
GPU boosted software is employed in advanced imaging
diagnostic such as tumor image segmentation Zhao at al.
(2018); Zhang at al. (2017) or in real-time simulation
of surgical operations Joldes, Wittek and Miller (2010);
Kubisch, Tietjen and Preim (2010). Planning anti-cancer
treatment, especially for radiation therapy, is the following
domain where application of graphic processors is fully
justified. Mathematical models described by the partial
differential equations (PDE) are integrated for calculating
required doses of radiation for given initial conditions
such as tumor size, its type and location Mariappan at
al. (2017); Yepes, Newhauser and Elay (2016). However,
despite the high computational complexity of these models,
they are still too simple to be used as a prognostic tool in
clinical oncology. To this end, the response of tumor and its
environment on the applied therapy should be modeled and
simulated.
During more than forty years of history of computational
oncology, dozens of mathematical and computational
models were developed to model cancer evolution, i.e.,
its progression, regression/remission and recurrence (see,
e.g., Marias at al. (2011); Deisboeck at al. (2011); Cristini
and Lowengrub (2010); Wodarz and Komarova (2014);
Rieger, Fredrich and Welter (2016); Lima at al. (2016)).
Mostly, they are focused on the processes stimulating tumor
growth, and have only purely scientific character, i.e.,
they were developed to better understand the intrinsically
complex process of oncogenesis. Due to multiscale cancer
dynamics, which spans from molecular to tissue levels under
continuous and unpredictable influence of the environment,
the application of the most advanced multi-scale tumor
models Marias at al. (2011); Deisboeck at al. (2011)
to prognostic and personalized oncology is not realistic.
Mainly, due to many principal conceptual and computational
problems Dzwinel at al. (2016); Dzwinel, Klusek and
Paszynski (2017) including overfitting, ill-conditioning and
computational complexity.
In our opinion, the future of computer simulation of
cancer dynamics in predictive oncology can look like climate
modeling and weather forecast, where the prognoses are
elaborated in a very intuitive prediction/correction scheme.
The numerical simulations could be verified continually
by incoming data and reinforced by data-based models.
However, unlike in the climate/weather forecast, data coming
from tumor observation are scarce relative to the number of
free model parameters. Moreover, matching even those the
most important to data is not a straightforward procedure
as it is in climate/weather modeling. Therefore, to avoid the
overfitting and simultaneously decrease the time complexity,
AGH University of Science and Technology, Department of Computer
Science, Kraków, Poland
Corresponding author:
Adrian Kłusek, AGH University of Science and Technology, Department
of Computer Science, Kraków, Poland
Email: klusek@agh.edu.pl
Prepared using sagej.cls [Version: 2016/06/24 v1.10]

2Journal Title XX(X)
the cancer models used for elaborating prognoses should
be as simple as possible, i.e., the set of model parameters
should be small to make data adaptation procedures (such as
pABC Jagiella at al. (2017) and supermodeling Duane at al.
(2018); Dzwinel, Klusek and Vasilyev (2016)) reliable and
affordable computationally. On the other hand, toy models of
cancer dynamics Wodarz and Komarova (2014); Ribba at al.
(2012,2014); Park (2017), which neglect spatial coordinates,
can be useless producing very unreliable prognoses such as
in Ribba at al. (2012). So, the following question should be
raised: How complex 3D model of tumor dynamics can be
considered nowadays as a predictive platform, which is both
non-trivial and, simultaneously, computationally efficient?
In this paper we present multi-GPU CUDA implemen-
tation of 3-D tumor model of cancer dynamics, which
employs classical numerical engine based on finite difference
method (FDM). We demonstrate its high efficiency for both
a standard setup representing a homogeneous tissue and a
setup which is more topologically complex and mimics the
skin structure. We show that our implementation is supe-
rior over other GPU implementations of discrete/continuous
tumor models Worecki and Wcisło (2012); Wcisło at al.
(2013); Wang et al. (2015); Zhang at al. (2011); Wang at
al. (2015) and multi-thread CPU models based on more
complex numerical engines Ło´
s at al. (2017). We conclude
what large tumor simulations can be currently performed by
using mid-range multi-GPU servers in a reasonable time.
Tumor model
Because anti-cancer therapies usually are implemented for
solid tumors with diameter of order of centimeters, we con-
sider here single-phase, heterogeneous continuous/discrete
model of cancer. The GPU implementation of discrete/con-
tinuous models, such as particle automata (PAM) Worecki
and Wcisło (2012); Wcisło at al. (2013) and agent-base
models Jagiella at al. (2017); Wang et al. (2015); Zhang at
al. (2011); Wang at al. (2015), where the tissue is represented
by the cell-cell interactions while oxygen, TAF (tumor angio-
genic factors) and nutrients concentrations are calculated by
integrating diffusion equations, allow us for simulating rather
small tumors of a mesoscopic scale, i.e., a few millimeters in
diameter. As shown in Jagiella at al. (2017), the evolution
of N= 106cells (tumor and healthy ones) in 3D involves
3-4 days of CPU time (5-6 minutes on HPC cluster with
1000 CPU cores Jagiella at al. (2017)). Thus, the data
adaptation process needs a month of calculations on the high
performance computing system. Moreover, for larger tumor
sizes Lthe required computational power increases as L3.
Our continuous/discrete tumor model consists of:
1. the continuous mathematical model, describing the
evolution of cancer tissue and other important tumor
factors (oxygen, angiogenic factors (TAF), fibronectin,
ECM etc.) by means of mainly diffusion-reaction
partial differential equations (DR-PDE);
2. a predefined discrete graph of vessels and initial
setup representing spatial initial conditions of tumor
dynamics;
3. set of rules coupling continuous model with the
processes of vessels remodeling (translation, growth,
dilation and collapse).
Mathematical framework
The main equations describing the tumor model come mainly
from Deisboeck at al. (2011); Cristini and Lowengrub
(2010); Wodarz and Komarova (2014); Rieger, Fredrich
and Welter (2016); Welter and Rieger (2010); Kim, Gillies
and Rejniak (2013). In Table 1 we collect the detailed
description of parameters and their values on the basis of
the tumor models presented in Welter and Rieger (2010);
Ramis-Conde, Chaplain and Anderson (2008); Chaplain,
McDougall and Anderson (2006); Barillot at al. (2012);
Manning (2013).
Tumor cell density The principal density field brepresents
tumor cell density, which is from [0,2] interval, where 0
means lack of cancer cells and bM= 2 is the maximum
cancer cell density. Its dynamics is governed by the following
diffusion equation:
∂b
∂t =−∇ · J+b−+b++bTh(1)
where Jis the tumor cell flux, and b+,b−describe sources
and sinks. They correspond to cell proliferation (mitosis)
and cell death (necrosis), respectively. The value of bTh
accounts for the effects of the anti-cancer drug action. Tumor
cell flux is stimulated by tumor pressure and interactions
with degenerated extracellular matrix (ECM), while sources
and sinks are governed by oxygen supply. Assuming that P
denotes the tumor pressure given by P=θ(b), where θis a
function increasing linearly from θ(bN) = 0 to θ(bM)=1,
i.e.
θ(b) = (0for b < bN
b−bN
bM−bNfor bN≤b≤bM
(2)
and
J=−Dbb(∇P+rb∇A)(3)
where Ais degraded ECM density (see Extracellular
matrix) and Db,rbare the constants.
Tumor cell proliferation is switched on when oxygen level
oexceeds oprol, while necrosis occurs when o<odeath.
Corresponding terms are given by:
b+=b
Tprol 1 + τbA
τbA+ 1Pb1−b
bMfor o>oprol
b−=−b
Tdeath for o<odeath
bTh=−γThbTh
(4)
Extracellular matrix (ECM) The corresponding concentra-
tions of normal and degraded ECM, denoted by Mand A
respectively, are described by the following PDEs:
∂M
∂t =−βMM b
∂A
∂t =γAM b +χaA∆A−γoAA
(5)
Prepared using sagej.cls

3
Figure 1. The initial setup for continuous/discrete melanoma model. In (a) the skin tissue layers are depicted from the top to the
bottom: stratum corneum,stratum spinosum,basement membrane,dermis and hypodermis. The zoom-in of initial vasculature is
shown in (b). The lateral (c) and aerial (d) views of melanoma blob are presented. The color of tissue represents the density of tumor
cells (red is the highest) while the color of blood vessels their diameters (red is the most dilated). The black necrotic center is clearly
seen in (d).
Tumor angiogenic factors (TAF) Tumor cells not suffi-
ciently oxygenated produce various proteins (TAF), which
stimulate the evolution of the vasculature. The concentration
of TAF cis given by the following equation:
∂c
∂t =χc∆c−γco c +c+(6)
where
c+=b·((1.0 + tanh((odeath −o)/0.1))/2.0) ·(1.0−c)
(7)
and zero otherwise.
Oxygen concentration The concentration of oxygen ois
calculated integrating the following equation:
∂o
∂t =αo∆o−(b+θo)γoo+δo·o2source
·omax ·((1.0 + tanh((omax −res)/2.0))/2.0)
(8)
Drug concentration The concentration of anti-cancer drugs
This described as follows Kim, Gillies and Rejniak (2013):
∂Th
∂t =αTh∆Th+κTh·o2source
·((1.0 + tanh((1.0−Th)/0.1))/2.0)
−Th·ΓTh·b−βTh·Th
(9)
Necrotic core When the oxygen concentration decreases,
the cancer cells die forming necrotic core Nec.
∂N ec
∂t =−γN ec ∗N ec (10)
We assume that:
i f ( ( T>Tdeath )and (b > bN)and (Nec ==
0.0) ) {
Nec = b ;
b = 0 . 0 ;
}
where the value of Tis the time the cells spent in the
absence of oxygen. The necrotic cells are removed from the
tissue according to the equation:
Time of cells in absence of oxygen In order to calculate
the time the cells spent in the absence of oxygen, we use the
following procedure:
i f (o<odeath ) {
i f (c > 0.0) {
T += d t
}else{
T−= d t
}
}else{
T−= d t
}
i f ( T < 0 . 0 ) {
T = 0 . 0
}
Necrotic, quiescent and proliferative cells number For cal-
culation the statistics of proliferative proliferativeC ells,
necrotic necroticCells and quiescent cells quiescentC ells
we use the following procedure:
for (int j =0 ; j < p o i n t s ; ++ j ) {
CC = b [ j ] ;
NEC = Nec [ j ] ;
i f ( ( CC > bN) &&(NEC == 0. 0 f ) ) {
proliferativeCells += 1;
}e l s e i f (NEC > 0.0) {
necroticCells += 1;
}e l s e i f ( ( CC > 0.1)&&(CC < bN)
&&(NEC == 0 . 0 f ) ) {
quiescentCells += 1;
}
}
Vascular network
We simulate the dynamics of tumor in two layouts
mimicking two different environments: homogeneous and
heterogeneous tissues. The former represents a general
metaphor of cancer while the latter one mimics the skin
structure. As shown in Figure 1, the skin model is greatly
simplified comparing to the real one. It consists of five
Prepared using sagej.cls

4Journal Title XX(X)
Figure 2. The plots, which demonstrate deviations of the tumor size (in the number of tumor cells) with the change of the respective
parameters (in the interval of 10% of their reference value).
layers representing stratum corneum,stratum spinosum,
basement membrane,dermis and hypodermis. Different
diffusion coefficients of tumor cells are assumed in the
various skin layers (see Table 1).
For the two types of tissue (cancer growth environments)
we have matched the model parameters and the vasculature
in such a way that the tissue is initially well oxygenated. We
assume that the vessel network is made of short segments
with length of the order of the mesh size. Due to very
specific way that the blood network in skin tissue is arranged
(see Figure 1), comparing to a homogeneous tissue (see
Łazarz (2016)), the algorithm for blood network creation
adds the various layers of vessels in subsequent iterations.
All of them are made of a series of line segments of the
length equal to a "segment_length" parameter. The first
layer consists of horizontal "base" vessels, where parallel
artery-vein pairs are layered at the bottom of the model
(see Figure 1b). Their thickness, length, and the number
of pairs are defined by respective model parameters. The
bifurcation points for capillaries going towards the surface
of the skin are chosen randomly for each base vessel. At this
point there are two queues created. One for veins and one
for arteries. They will serve as the places to store endings
of vessels during the blood network creation process. The
splitting points from the base vessels are added subsequently
to the queue. The middle blood network layer is created
by extracting all of the vessel endings currently stored in
queues and followed by two steps: by adding randomly a
new "non-splitting" vessel segment and then by creating two
new vessel segments starting at extracted segment tip. The
ending of the new segment (or segments) is always chosen
exactly "segment_length" above its starting point and moved
randomly in other directions by a given "max_curvature"
parameter. This way the new segment tip (or tips) is added to
the queue. The number of middle layers is defined by "levels"
parameter. The thickness of a vessel in each layer is inversely
proportional to its number, ranging from "max thickness" to
"min thickness". The top network layer defines connections
between the blood vessels. The vessels endings are taken
subsequently from the bigger queue in pairs and connected
together. The junction is placed always in the middle of
vessels and is added to the queue after a segment creation.
The last step of the network formation consists in connecting
veins with arteries. One segment tip is chosen from each
of the queues and the vessel is created between them in a
similar fashion as non-splitting vessels in the middle layers.
As shown in Figure 8a, the melanoma is initiated at the center
of the stratum spinosum skin layer just above the basement
membrane seen in Figure 1a.
Unlike melanoma setup, the topology of blood vascular
network for a homogeneous cancer resembles that from
Łazarz (2016). Instead of many tissue layers only one
"layer" is defined. The tumor is initiated at the center of the
simulated space in the thicket of blood vessels (see Figure
5a). The blood network models were prepared by using
UnityTM computer games engine.
Vasculature remodeling
In our continuous/discrete model, the concentration field
of cancerous tissue interacts with the discrete vasculature.
Blood vessels network remodeling and its spatio-temporal
evolution is the crucial process influencing tumor prolifer-
ation, its remission and recurrence. It is well known (see
e.g. Rieger, Fredrich and Welter (2016); Łazarz (2016)) that
the network of blood vessels inside and close to the tumor
tissue is dynamically unstable. The interactions between the
processes of new vessels creation, due to angiogenesis and
the shearing forces coming from purely mechanical stress,
stimulated by increasing tumor mass result in unsystematic
blood flow in the vasculature and, consequently, influence
the topology of the blood vessels network. Its structure
becomes very complex and irregular due to continual process
of vessels reshaping, changing their locations, dilation, their
decay and collapsing. Vessels remodeling directly influences
the growth conditions of the cancerous tissue, making its
dynamics even more chaotic. Thus the tissue and vasculature
are coupled in an intrinsically complex way.
During the simulation we have to recompute the blood
flow, new vessel creation and destruction in the graph of
vessels every a few hundred of timesteps (240 in simulations
presented below for given ∆tfrom Table 1) by adapting
the concepts presented in Marias at al. (2011) and Cristini
and Lowengrub (2010). This is a reasonable assumption
knowing that the process of vessels remodeling is much
slower than cancer cells diffusion and advection. The rest
of processes are recomputed in every time step. We assume
that both the oxygen and hematocrit concentrations in blood
do not change along the vessels and the oxygen supply is
proportional to the blood flow rate.
Summing up, both in our discrete and continuous/discrete
models of melanoma dynamics, the blood network remodel-
ing consists of the following processes.
Prepared using sagej.cls

5
Figure 3. The timings for AWESUMM and IGA-FEM solvers, respectively, obtained for simulation of 1000 timesteps of the tumor
dynamics in 3D. a) AWESUMM was ran for 5·104collocation points. The effect of cancer simulation with and without vessels
remodeling part is clearly seen. b) The timings for IGA-FEM solver with increasing number of cores and for various number of
elements.
Sprout initiation: A new vessel segment can be added
with a given probability ∆vessel_add(= 0.01) at any
location on the network where TAF concentration is greater
than ∆taf_conc(= 0.00003).
Vessel collapse: A vessel segment can be removed with
a given probability tcoll_vess ∈[0,0.004] if its wall stability
is equal to zero and the wall shear stress is below a
given threshold. A sprout segment can be removed with the
probability tcoll_capi ∈[0,0.001].
Wall degeneration: The structural support provided
by the cell layers surrounding the endothelial cells is
represented by the wall stability parameter. For new vessels
and the original vasculature it is initialized with the wall
diameter of healthy vessels. For vessels inside the tumor its
value decreases to zero at a constant rate ∆vessel_degr(=
0.05).
Vessel dilation: The vessel radius rincreases at the
constant rate rc= (0.4µm/h)if r < rmax[25µm]and if the
average growth factor concentration in the segment vicinity
and the time it spent in the contact with tumor are greater
than given threshold values respectively vessel_cmin(=
0.01) and vessel_tmin(= 40h).
Sprout degradation: If the sprout newly formed is
located in tumor more than sprout_tmax(= 200h)it is
going to be removed.
Description of parameters
In this section, in Table 1, we summarize all the parameters
of our model and their sensitivity.
Sensitivity analysis
Because the tumor model requires estimation of many
parameters, we performed the analysis of their sensitivity.
Before, however, we have matched adequately the numerical
parameters (such as ∆tand ∆x) to make their influence
on simulation results negligible low. We focused on
the simulation of tumor proliferation thus the equations
representing anti-cancer treatment and its influence on tumor
dynamics were neglected in sensitivity analysis.
We ran simulations shifting subsequently every model
parameter from 1 up to 10 percent below and above
its reference value, simultaneously, keeping all the others
unchanged. After 104timesteps we measured the overall
volume of the tumor. Sensitivity analysis is very demanding
Table 1. Continuous model parameters with description and
sensitivity. The sensitivity is marked by colours: RED - the
highest, BLUE - medium, GREEN - low. Parameters are
dimensionless.
Symbol Value Description
bm0 min tumor cell density
bM2 max tumor cell density
bN1 normal tumor cell density
Db
0.00002
0.0000415
0.00000083
0.000083
0.00002
tumor cell diffusion rate
rb0.3tumor cells chemoattractant
sensitivity
oprol 10 tumor proliferation threshold
odeath 2tumor cell hypoxia threshold
Tprol 10 tumor cell proliferation time
Tdeath 100 tumor cell survival time
Pb0.001 maximum stimulated
mitosis rate
τb0.5 instantaneous reaction rate
γTh0.4 threatment influence
βM0.0625 ECM decay rate
γA0.032 production rate of attractants
χaA 0.000625 decay rate of digested ECM
γoA 0.000625 diffusion rate of digested ECM
χc0.0000555 TAF diffusion rate
γc0.01 TAF decay rate
αo0.0000555 oxygen diffusion rate
θo0.1 oxygen consumption
by normal cells
γo0.01 oxygen consumption rate
δo0.4 oxygen delivery rate
omax 60 maximal oxygen concentration
αTh0.00008 anti-cancer drug diffusion rate
κTh0.6 production rate of treatment
ΓTh0.00005 decay rate of drug
βTh0.00001 deactivation of drug
γNec 0.00001 decay rate of necrotic core
∆t0.1h time step
∆x20µm space step
computationally because we had to ran 20 simulations
for every parameter. Therefore, to shorten the total
Prepared using sagej.cls

6Journal Title XX(X)
computational time we performed the calculations for 2D
tumor model Ło´
s at al. (2017). The parameters sensitivity
produces rather qualitative results, thus they should not
deviate greatly from those obtained in three dimensions. The
detailed results are collected in Siwik et al. (2018). In Table
1 we categorized the changes in tumor size as low (<2%),
medium (2-4%) and high (>4-10%). It appears that the tumor
growth is the most sensitive only on two parameters (oprol
and Tprol) influencing directly the cancer cells proliferation
and survival time stimulated by their response on the oxygen
supply. This result is very intuitive. However, it is interesting
that the most of the parameters (with medium sensitivity)
appeared to be very variable, not revealing any functional
dependence of the size of tumor on its value (such as odeath
in Figure 2a). The exceptions are Tprol and oprol . They show
clear monotonic relationship of the tumor size and parameter
value (such as Tprol in Figure 2b). The simulations mimics
the initial – avascular – tumor growth (approximately up
to 1 mm) so the access to diffusive oxygen is a critical
factor of its evolution. On the following – vascular –
growth stages other parameters can be more sensitive, e.g.,
those influencing remodeling of the vascular network and
stimulating angiogenesis. Anyway, the results of sensitivity
analysis show that despite the substantial number of the
model parameters, the data adaptation procedure may focus
only on a few, the most sensitive, parameters on each
stage of tumor evolution. However, to this end, the tumor
model should not only mimic the main properties of cancer
dynamics but it has to do it in a realistic computational time.
Computational model
The described continuous/discrete model is numerically
integrated by using classical FDM on the regular mesh
consisting of M= 1.56 ·107,(250 ×250 ×250) to M=
0.5·109,(750 ×750 ×750) nodes. The parabolic PDEs
were solved by using explicit Runge-Kutta 2nd order
scheme. The continuous/discrete melanoma model was run
on the two layouts described above corresponding to a
fragment of tissue placed in the cubic box of size 5mm
up to 13mm (the simulated skin tissue is 1 mm smaller in
height due to an empty space above the skin). The Dirichlet
boundary conditions were applied. The timestep in the real
time-scale is about ∆t= 6 minutes. Full simulation, i.e.,
when the tumor occupies 2/3of the computational box,
requires about 2·104timesteps (for M= 1.56 ·107case).
The values of numerical parameters are collected at the end
of Table 1.
The numerical model was implemented in C++ in
CUDATM environment. The simulations were performed
on a single node of the ZEUS GPGPU cluster (ACK
CYFRONET, Kraków) equipped with 20 computational
nodes in total. Each node consists of two Intel Xeon X5645
(6 cores), 96 GB RAM and is associated with 8 Nvidia R

TeslaTM M2090 (512 cores, 6GB GDDR5) boards. We also
ran our code on one node of the second front-end server -
PROMETHEUS - which consists of two Nvidia Tesla K40
XL GPU cards and two Intel Xeon E5-2680v3 CPUs (24
threads).
Listing 1: A fragment of simplified CUDA code, which
depicts the redistribution of parallel numerical integration of
equations describing continuous/discrete tumor model.
__global__
v o i d funA ( float∗t a b l e C a n c e T 0 , d l o a t d t ,
float∗tablceCancerT1 , float dx ,
int dimX , i n t dimY , i n t di mZ ) {
/ / C a l c u l a t i o n o f t h e ( x , y , z ) p o s i t i o n
i n t x = t h r e a d I d x . x + b lo c k I d x . x ∗blockDim . x ;
i n t y = t h r e a d I d x . y + b lo c k I d x . y ∗blockDim . y ;
i n t z= t h r e a d I d x . z + b lo c k I d x . z ∗blockDim . z ;
/ / C he c k i n g di m e n s i o n s
i f ( ( x>=dimX ) | | ( y >=dimY ) | | ( z >=dimZ ) ) {
r e t u r n ;
}
/ / R e ad i n g c u r r e n t v a l u e o f A f o r t i m e T
float A = t a b l e C a n c e r T 0 [ t r 3D t o 1 D ( x , y , z ,
dimX , dimY , dimZ ) ] ;
/ / C a l c u l a t i o n s on va l u e A
/ / S e t t h e new va l u e f o r t i m e T +1
t a b l c e C a n c e r T 1 ( t r3 D t o 1D ( x , y , z ,
dimX , dimY , dimZ ) ) = A ;
}
In Listing 1 we present a fragment of simplified CUDA
code used for parallel numerical integration of differential
equations describing continuous/discrete melanoma model.
It shows how, in an optimal way, redistribute the calculations
on the mesh onto GPU threads and blocks.
The most difficult problem in heterogeneous continu-
ous/discrete and discrete/continuous models Dzwinel at al.
(2016,2018) is the problem of coupling their continuous and
discrete components. In case of tumor dynamics we show
in Dzwinel at al. (2016); Worecki and Wcisło (2012) that
the vessels remodeling is the most intensive computationally
part of the particle based discrete/continuous PAM model.
Moreover, it is the model component, which decreases dras-
tically the overall gain in efficiency expected in case of
its implementation in GPU/CUDA environment. Similarly,
we demonstrated in Dzwinel at al. (2018) that coupling the
continuous tumor model with blood vessels can represent
the serious issue, especially, for ready to run numerical
engines intended for modeling classical fluid dynamics or
CAD/CAM systems. We consider this problem in a more
explicit way in the following section.
Unlike the solvers mentioned above, our model is
integrated by using the direct numerical schemes and it
is coded from the scratch in C++ and CUDA. So we
are able to test how the architectural issues related to
CPU/GPU cooperation influence the efficiency of coupling
of continuous and discrete model components. To this end,
we have developed two versions of the vessels remodeling
procedure. In the first one, most the functions which are
the most demanding computationally, are calculated on GPU
and the rest on CPU, while in the second, the whole
vessels remodeling process is calculating on CPU. The
corresponding pseudocodes are presented in Table 2 and
commented briefly in its caption.
Prepared using sagej.cls

7
Table 2. For the first version of our code, the co-functions invoked in calculateStep1() procedure (responsible for the most
computationally intensive fragments of the vessels remodeling simulation) are computed on GPU whereas those calculated in the
procedure calculateStep2() are executed on CPU. In the second version, all calculations in calculateStep1() and calculateStep2()
procedures are executed on CPU.
First version: C++ + CUDA Second version: C++
v oi d c a l c u l a t e S t e p 1 ( ) {
cpNumberOfVesDevToHost ( ) ;
i nt n = numberOfVes ;
v e s D e g r a d a t i o n <<< n > > > ( ) ;
v e s D i l a t i o n < <<n > > > () ;
u pd a te T im e In T um o r << <n > > > () ;
}
v oi d c a l c u l a t e S t e p 2 ( ) {
cpVesDevToHost () ;
v e s C o l l a p s e ( ) ;
sproutCollapse () ;
sproutInitiation () ;
a n a s t o m o s e S p r o u t s ( s i m ) ;
c a l c u l a t e F l o w ( s im ) ;
cpVesHostToDev () ;
}
f or ( i n t i : i t e r a t i o n s ) {
calculateStep1 () ;
i f ( i %24 0 == 0 ) {
calculateStep2 () ;
}
}
v oi d c a l c u l a t e S t e p 1 ( ) {
v e s D e g r a d a t i o n ( ) ;
vesDilation ();
updateTimeInTumor ();
}
v oi d c a l c u l a t e S t e p 2 ( ) {
v e s C o l l a p s e ( ) ;
sproutCollapse ();
sproutInitiation ();
a n a s t o m o s e S p r o u t s ( s i m ) ;
c a l c u l a t e F l o w ( s im ) ;
}
f or ( i n t i : i t e r a t i o n s ) {
calculateStep1 ();
i f ( i %24 0 == 0 ) {
calculateStep2 ();
}
}
Timings and results of cancer dynamics
simulation
Timings
Apart from the FDM/GPU solver, we tested two additional
numerical engines:
a) AWESUMM (Adaptive Wavelet Environment for
in Silico Universal Multiscale Modeling), based
on the finite-difference second-generation wavelet
collocation method invented by Vasilyev et al. (see e.g.
Vasilyev and Kevlahan (2005));
b) IGA-FEM (Isogeometric Finite Element Analysis) L2
projection solver (se e.g. Ło´
s at al. (2017)), which
utilizes B-spline and NURBS basis functions.
The two solvers are parallel and realized in MPI
and multi-thread architectures, respectively. Despite their
high efficiency in simulating continuous systems, their
direct application to heterogeneous continuous/discrete
3D simulations failed, due to extremely computationally
demanding procedure of vessels remodeling. In Figure 3a we
present timings from simulation of 1000 timesteps of tumor
growth by using AWESUMM solver. Despite this code is
parallel and uses MPI interface, even if ran on 16 cores, it is
still very slow having in mind that the full simulation requires
at least 2·104timesteps. Moreover, its low resolution
(5·104collocation points) cannot be sufficient assuming
denser vascularization when the resolution of the simulated
system is determined by the shortest distance between the
vessels. Even worse timings were obtained for the IGA-
FAM solver. As shown in Figure 3b, its good scaling
properties with the number of cores resulting in not bad
efficiency in 2D simulation of cancer Ło´
s at al. (2017),
cannot mitigate its dramatically low performance in three
dimensions. Currently, we are working on this problem
trying to fix it and implement the IGA-FAM solver in multi-
GPU/CUDA environment.
In Table 3 we present the timings obtained for tumor
dynamics simulation by proposed FDM/GPU engine with
increasing number of GPUs. Two versions of the cancer
model were tested, i.e., the continuous/discrete with
vessels and purely continuous one without blood vessels.
In this second case we consider only initial, diffusive
stage of cancer growth. Moreover, in the case of the
continuous/discrete model we have two variants of vessels
remodeling implementation shown in Table 2. In the first
one ("First version" in Table 2), only vessels collapsing and
their initialization due to angiogenesis are calculated on CPU
(every 240 timesteps) while vessels degradation, dilation
and translation is computed on GPU every timestep. In the
second ("Second version" in Table 2), the whole remodeling
process is realized on CPU. In Table 3 we show the timings
for various sizes (computational mesh resolutions) of the
tumor model.
Prepared using sagej.cls

8Journal Title XX(X)
Table 3. Comparison of timings obtained for 1000 timesteps of tumor simulation with various number of GPUs for two
implementations of the vessels remodeling process (see Table 2). The calculations were performed for one node of Zeus (upper
row) and Prometheus (lower row) high-end servers. The left panel shows timings for the model with discrete blood vessels, while the
right one without. The table below, presents the comparison of vessel remodeling time relative to the whole simulation time for faster
(on the left) and slower (on the right) multi-GPU implementations of the tumor model for Zeus high-end server.
1 GPU 2 GPU 4 GPU 6 GPU 8 GPU
250x250x250 0.6% | 42.65% 1.4% | 57.6% 3.3% | 66.7% 5% | 67.8% 5.5% | 63.6%
500x500x500 x 0.2% | 29.5% 0.7% | 43.1% 1.1% | 48.8% 1.4% | 40%
750x750x750 x x x 0.5% | 45.4% 2.3% | 46.1%
Figure 4. The temporal evolution of the tumor size (proliferative, quiescent and necrotic cells) modeled by a) simplified 0D model of
glioma Ribba at al. (2012), b) our 3D tumor model.
As shown in Table 3 the timings on older ZEUS high-
end server (GPU - M2090 with 512 cores and memory
throughput 177GB/s) for continuous and continuous/discrete
models are very similar for the first variant of the code
("First version") and for larger systems. The table in Table
3, positioned below the plots, presents the comparison of
vessel remodeling time relative to the whole simulation
time for the "First version" (the table cells on the left) and
"Second version" (the table cells on the right) variants of
vessels remodeling implementation. However, this difference
is not visible for timings obtained on modern Prometheus
server (GPU - K40 with 2688 cores and memory throughput
288GB/s). The two plots look exactly the same. The distinct
speed up of the "First version" variant over the "Second
version" one is visible for two smallest systems (250 ×
250 ×250 and 500 ×500 ×500) while for the largest
(750 ×750 ×750) it is negligible small. It means that
the transmission rates from CPU to GPU memory for
Prometheus are much faster than for ZEUS.
It is also clearly seen that the system simulated
on 250 ×250 ×250 mesh, is too small to exploit the
multiple GPU parallelism, and some speedup is observed
for only two GPUs. Further increase of the speedup
with the number of GPUs is canceled due to increasing
communication/computation ratio. By increasing the system
size 8 times (500 ×500 ×500) and comparing to the
previous one its scalability (due to better load ballancing)
with the number of GPUs slightly improves on ZEUS GPUs
Prepared using sagej.cls

9
Figure 5. The snapshots from simulations of the continuous/discrete model of tumor dynamics in a homogeneous tissue. The
system represents the tissue fragment 5mm x 5mm x 5mm (M= 1.56 ·107mesh nodes). The darker color of the cluster depicts
greater density of tumor cells.
what results in the speedup equal to 2 on 6 GPUs. Similar
speed up was obtained on only two much faster Prometheus
GPUs. Moreover, due to greater memory requirements for
simulation of the system of that size, at least 2 GPUs are
needed for ZEUS to run the simulation. Consequently, by
further increase of the mesh resolution (750 ×750 ×750
nodes, i.e., the system is 27 times greater), the required
minimal number of GPUs (ZEUS) increases to 6. For this
mesh size and 8 GPUs, the relative speedup in instructions
per cycle, comparing to 250 ×250 ×250 system, increases
to about 2. Meanwhile, no relative speedup is measured for
Prometheus, due to very fast communication inside GPU
memory subsystems and between GPU and CPU memory
systems. Anyway, the total computational time for the typical
simulation of cancer growth to the diameter equal to 5
mm (250 ×250 ×250,2·104timesteps) run on 4 GPU
boards is approximately 15 minutes. For largest simulation
(750 ×750 ×750,4·104timesteps, tumor size about 1.3
cm) the computational time is 5 hours on 8 GPUs (Zeus) and
about 6 on one Prometheus node (2 GPUs). However, from
Table 3 it is clearly seen that, the possibility to scale up the
computations with increasing memory demands, maintaining
constant or even greater then the relative (in instructions per
cycle) GPU performance, is the greatest benefit from multi-
GPU computations.
Results of cancer dynamics simulations
We have performed the simulations of tumor dynamics
for two 3D layouts mimicking a homogeneous tissue and
heterogeneous skin structure. The first one can be used for
simulating, e.g., the glioma cancer progression while the
second one the melanoma evolution. As shown in Ribba at
al. (2012), glioma growth can be modeled by an expanded
0D (structureless) predator-pray paradigm described by the
set of a few ODEs. The model assumes that the cancer tissue
consists of proliferative and quiescent (dormant, and not able
to divide) tumor cells. The cell becomes quiescent if it avoids
necrosis, i.e., cell death, due to deep hypoxia and/or the high
pressure. In Figure 4a, we show the dynamics of glioma
growth, obtained for the predator-pray model assuming that
the maximal MTD (mean tumor diameter) is equal to 32 mm.
The model parameters were taken from Ribba at al. (2012).
In Figure 5 we present the snapshots from 3D simulations
for the first, homogeneous, layout, which mimics a soft
tissue with vascularization simulated by using the blood
network model from Wcisło at al. (2013). After a proper
Prepared using sagej.cls

10 Journal Title XX(X)
Figure 6. The snapshots from simulations of the continuous/discrete model of tumor dynamics in a homogeneous tissue. The
computational box represents the tissue fragment 13mm x13mm x 13mm of size (M= 0.5·109mesh nodes) after 4·104timesteps.
The darker color of the spots depicts the greater density of tumor cells (a). The vessels inside tumor undergo dilation (represented
by red and thick capillaries) and collapse (b).
adaptation of the model parameters Friesboes at al. (2007)
the layout could mimic also glioma evolution. The growth
of cancer cells population is presented in Figure 4b. It stops
when the maximal tumor cluster size (MTD) reaches 5 mm
(compare Figure 4b to Figure 4a). In the avascular stage
of growth (up to 104timesteps) the tumor pushes out the
vessels decreasing oxygen concentration in the center of
the spherical cluster of cancer cells. Some cells inside the
spheroid become hypoxic and after some time many of
them, arrested in the tumor center, die creating a necrotic
core (see Figure 1d). However, some of the cells escape
hypoxia and become dormant (quiescent) due to mechanical
tissue/vessels remodeling and the process of angiogenesis.
Consequently, as shown in the last snapshots from Figure
4, the surface of spheroid become uneven being controlled
by the chaotic coupled processes of vessels remodeling and
tissue dynamics. The spots with a greater density of tumor
cells have simultaneously denser vasculature (thus they are
better oxygenated) than those where it is sparse or lacking.
This results of vessels remodeling can be seen clearly in
Figure 6. As shown in Figure 6b, the vascularization inside
the tumor is very sparse and blood capillaries become dilated
comparing to those outside. Because of low blood pressure
and limited perfusion they collapse finally.
Our cancer model is able to simulate also the effect of anti-
cancer drugs influence on the tumor dynamics. This effect
can be clearly seen in Figure 7. The necrosis encompasses
almost all tumor mass, except a small spots of dormant
cells, which might be potentially a new sources of the cancer
recurrence.
The second layout (see Figure 1) defines the initial
conditions for skin cancer simulation. The modeled tissue
is not homogeneous one and, as described in Section 2.2,
consists of a few skin layers and very specific structure
of the blood network. Moreover, the free surface of skin
has to be modeled, to simulate protrusion of cancerous
tissue. In Figure 8, we can follow the development of
the cone shaped nodular melanoma. However, as shown
in Dzwinel, Klusek and Vasilyev (2016); Kłusek, Dzwinel
and Dudek (2016); Dzwinel at al. (2018), the changes
of the parameter representing the mechanical interactions
between tumor cells in various levels of the skin tissue, can
produce the melanoma shapes similar to that observed in
the micrographs. Particularly, the hardness of the basement
membrane, which is located between stratum spinosum and
dermisis, is responsible for the vertical growth of the tumor.
The possibility of simulation of main scenarios of melanoma
proliferation were demonstrated in Dzwinel, Klusek and
Vasilyev (2016); Kłusek, Dzwinel and Dudek (2016) by
using the supermodeling paradigm Duane at al. (2018) where
the supermodel of cancer is represented by an ensemble of its
coupled sub-models.
What is important, we do not observe any distinct
differences in terms of computational time for the two
exemplary layouts. It means that the high efficiency of our
multi-GPU/CUDA cancer model does not depend on the
structural properties of the tumor growth environment.
Concluding remarks
Herein, we have presented briefly a new multi-GPU/CUDA
computer model for cancer simulation in three dimensions.
We demonstrate that the model, consisting of the continuous
mathematical description of tissue with diffusing compounds
(oxygen, TAF, anti-cancer drugs) and discrete representation
of blood network, is not a trivial one and, simultaneously,
very efficient computationally. It can simulate the evolution
of various types of tumors from an initial nodule to the tumor
of centimeter size, on a mid-range servers equipped with
multi-GPU boards.
We realize also that both the relatively large number
of parameters and still substantial computational time,
nowadays, disables using computationally demanding data
adaptation procedures. Consequently, direct application
of the model in predictive oncology is now unrealistic.
However, we demonstrate that it could be possible in the
nearest 5 years when, according to the Moore’s law, 200 -
400 times acceleration of computer power is expected.
Simultaneusly, to accelerate the clinical use of computer
modeling in personalized anti-cancer therapy, we are looking
Prepared using sagej.cls

11
Figure 7. The snapshots from simulations of the continuous/discrete model of tumor dynamics in a homogeneous tissue after
application of anti-tumor drugs. The computational box represents the tissue of 13mm x13mm x 13mm of size (M= 0.5·109mesh
nodes). The black color shows necrotic spots. In a) we demonstrate the initial tumor before anti-cancer treatment while in b) we can
see dying tumor remnants after injection of anti-cancer drugs.
Figure 8. The snapshots from simulations of the continuous/discrete model of melanoma dynamics in a heterogeneous skin tissue.
The system represents a tissue fragment 5mm x 5mm x 5mm (M= 1.56 ·107mesh nodes). The vessels inside tumor undergo
dilation and collapsing. The darker color of the cluster depicts the greater density of tumor cells.
for a faster method of data adaptation than those used
nowadays, e.g., pABC Bayesian approach (see e.g.Jagiella
at al. (2017)). We have demonstrated in Duane at al. (2018);
Kłusek, Dzwinel and Dudek (2016), that supermodeling
paradigm allows for reducing the number of parameters
for adaptation to only a few ones. It can be done by
coupling the sub-models through only the most important
dynamic variables (such as the concentration of tumor
cells) which are dependent on the most sensitive model
parameters. This way, only a few coupling coefficients could
be adapted instead of dozens of original model parameters.
We have been working also on developing a faster numerical
solver. To this end, we are working on implementation
of very perspective IGA-FEM isogeometric engine Ło´
s
at al. (2017) in multi-GPU/CUDA environment. We hope
that this research directions could make the modeling and
simulation of cancer a valuable tool in prognostic oncology
and planning personalized anti-cancer therapy.
Acknowledgement
This work was supported by National Science Center (NCN),
Poland Project no. 2016/21/B/ST6/01539 and partly (AK)
by the NCN project: 2013/10/M/ST6/00531. W thank dr
Leszek Siwik (AGH Department of Computer Science) for
Prepared using sagej.cls

12 Journal Title XX(X)
providing us with the results of sensitivity analysis and
corresponding figures.
References
Pratx, G., & Xing, L. (2011). GPU computing in medical physics:
A review. Medical physics, 38(5), 2685-2697.
Zhao, X., Wu, Y., Song, G., Li, Z., Zhang, Y., & Fan, Y. (2018).
A deep learning model integrating FCNNs and CRFs for brain
tumor segmentation. Medical image analysis, 43, 98-111.
Zhang, D., Zhu, X., Bifone, A., Gozzi, A., Capuani, S., & Palombo,
M. (2017). Efficient Parametric Imaging with GPU Computing.
Biophysical Journal, 112(3), 583a-584a.
Joldes, G. R., Wittek, A., & Miller, K. (2010). Real-time
nonlinear finite element computations on GPU–Application
to neurosurgical simulation. Computer methods in applied
mechanics and engineering, 199(49-52), 3305-3314.
Kubisch, C., Tietjen, C., & Preim, B. (2010). GPU-based smart
visibility techniques for tumor surgery planning. International
journal of computer assisted radiology and surgery, 5(6), 667-
678.
Mariappan, P., Weir, P., Flanagan, R., Voglreiter, P., Alhonnoro, T.,
Pollari, M., ... & Sequeiros, R. B. (2017). GPU-based RFA
simulation for minimally invasive cancer treatment of liver
tumours. International journal of computer assisted radiology
and surgery, 12(1), 59-68.
Yepes, P. P., Newhauser, W. D., & Eley, J. G. (2016). U.S. Patent
No. 9,495,513. Washington, DC: U.S. Patent and Trademark
Office.
Marias, K., Dionysiou, D., Sakkalis, V., Graf, N., Bohle, R. M.,
Coveney, P. V., ... & Clapworthy, G. (2011). Clinically driven
design of multi-scale cancer models: the ContraCancrum
project paradigm. Interface Focus, rsfs20100037.
Deisboeck, T. S., Wang, Z., Macklin, P., & Cristini, V. (2011).
Multiscale cancer modeling. Annual review of biomedical
engineering, 13, 127-155.
Cristini, V., Lowengrub, J. (2010). Multiscale modeling of
cancer: an integrated experimental and mathematical modeling
approach. Cambridge University Press, pp.278
Wodarz, D., and Komarova, N.L. (2014). Dynamics of cancer:
mathematical foundations of oncology. World Scientific,
pp.514
Rieger, H., Fredrich, T., & Welter, M. (2016). Physics of the tumor
vasculature: Theory and experiment. The European Physical
Journal Plus, 131(2), 1-24
Lima, E.A.B.F., Oden, J.T., Hormuth, D.A., Yankeelov, T.E., &
Almeida, R.C. (2016). Selection, calibration, and validation of
models of tumor growth. Mathematical Models and Methods
in Applied Sciences, 1-28
Dzwinel, W., Wcisło, R., Yuen, DA., Miller, S. (2016). PAM:
Particle Automata in modeling of multi-scale biological
systems, ACM Transactions on Modeling and Computer
Simulation, 26(3), A20:1-21
Dzwinel, W., Klusek, A., Paszynski, M. (2017), A concept of a
prognostic system for personalized anti-tumor therapy based
on supermodeling. Procedia of Computer Science, 108C,
1832–1841
Jagiella, N., Rickert, D., Theis, F. J., & Hasenauer, J. (2017).
Parallelization and high-performance computing enables
automated statistical inference of multi-scale models. Cell
systems, 4(2), 194-206.
Duane, G. S., Wiegerinck, W., Selten, F., Shen, M. L., &
Keenlyside, N. (2018). Supermodeling: Synchronization of
Alternative Dynamical Models of a Single Objective Process.
In Advances in Nonlinear Geosciences (pp. 101-121). Springer,
Cham.
Dzwinel, W., Klusek, A., Vasilyev, O.V. (2016). Supermodeling in
simulation of melanoma progression, Procedia of Computer
Science, 80, 999-1010
Ribba, B., Kaloshi, G., Peyre, M., Ricard, D., Calvez, V., Tod, M.,
... & Pallud, J. (2012). A tumor growth inhibition model for
low-grade glioma treated with chemotherapy or radiotherapy.
Clinical Cancer Research, 18(18), 5071-5080.
Ribba, B., Holford, N. H., Magni, P., Trocóniz, I., Gueorguieva,
I., Girard, P., ... & Friberg, L. E. (2014). A review of mixed-
effects models of tumor growth and effects of anticancer drug
treatment used in population analysis. CPT: pharmacometrics
& systems pharmacology, 3(5), 1-10.
Park, K. (2017). A review of modeling approaches to predict drug
response in clinical oncology. Yonsei medical journal, 58(1),
1-8.
Worecki, M., & Wcisło, R. (2012). GPU enhanced simulation of
angiogenesis. Computer Science, 13(1), 35.
Wcisło, R., Gosztyła, P., Dzwinel, W., Yuen, DA, Czech, W. (2013).
Interactive visualization tool for planning cancer treatment,
Lecture Notes in Earth System Sciences, in Yuen, D.; Wang,
J.; Johnsson, L.; Chi, C.-H.; Shi, Y. (Eds.), GPU Solutions
to Multi-scale Problems in Science and Engineering, Springer
Verlag, 607-637.
Wang, Z., Butner, J. D., Cristini, V., & Deisboeck, T. S. (2015).
Integrated PK-PD and agent-based modeling in oncology.
Journal of pharmacokinetics and pharmacodynamics, 42(2),
179-189.
Zhang, L., Jiang, B., Wu, Y., Strouthos, C., Sun, P. Z., Su, J.,
& Zhou, X. (2011). Developing a multiscale, multi-resolution
agent-based brain tumor model by graphics processing units.
Theoretical Biology and Medical Modelling, 8(1), 46.
Wang, Z., Butner, J. D., Kerketta, R., Cristini, V., & Deisboeck, T.
S. (2015, February). Simulating cancer growth with multiscale
agent-based modeling. In Seminars in cancer biology (Vol. 30,
pp. 70-78). Academic Press.
Ło´
s, M., Paszy´
nski, M., Kłusek, A., Dzwinel, W. (2017).
Application of fast isogeometric L2 projection solver for
tumor growth simulations. Computer Methods and Applied
Mechanics and Engineering, 316, 1257-1269
Welter, M. and Rieger, H. (2010). Physical determinants of vascular
network remodeling during tumor growth. The European
Physical Journal E. 33(2), 149-163
Ramis-Conde, I., Chaplain, M.A. and Anderson, A.R. (2008).
Mathematical modelling of cancer cell invasion of tissue.
Mathematical and Computer Modelling. 47(5), 533-545
Chaplain, M.A., McDougall, S.R. and Anderson, A.R.A. (2006).
Mathematical modeling of tumor-induced angiogenesis. Annu.
Rev. Biomed. Eng. 8, 233-257
Barillot,E., Calzone,L., Hupe, P., Vert, J.P., & Zinovyev,A. (2012).
Computational systems biology of cancer. CRC Press.
Manning, C.S. Heterogeneity in melanoma and the microenviron-
ment, PhD Thesis, University College London, January 2013.
Prepared using sagej.cls

13
Kim, M., Gillies, R. J.,& Rejniak, K. A. (2013). Current advances
in mathematical modeling of anti-cancer drug penetration into
tumor tissues. Frontiers in oncology, 3, 278.
Łazarz, R. (2016). Graph-based Framework for 3-D Vascular
Dynamics Simulation. Procedia Computer Science, 101, 416-
424.
Siwik, L., Ło´
s, M., Kłusek, A., Dzwinel, W., Paszy ´
nski, Tuning
tumor growth simulations, submitted to Summer Simulation
Multi-Conference, 2018
Vasilyev, O.V., & Kevlahan, N.K.R. (2005). An adaptive multilevel
wavelet collocation method for elliptic problems. Journal of
Computational Physics, 206(2), 412-431
Frieboes, H. B., Lowengrub, J. S., Wise, S., Zheng, X., Macklin,
P., Bearer, E. L., & Cristini, V. (2007). Computer simulation of
glioma growth and morphology. Neuroimage, 37, S59-S70
Kłusek, A., Dzwinel, W., & Dudek, A. Z. (2016). Simulation of
tumor necrosis in primary melanoma. In Proceedings of the
Summer Computer Simulation Conference (p. 55-61). Society
for Computer Simulation International
Dzwinel. W., Kłusek, A., Wcisło, R., Panuszewska, M., Topa,
P. (2018) Continuous and discrete models of melanoma
progression simulated in multi-GPU environment, Lecture
Notes in Computer Science, 10777, 505-518
Prepared using sagej.cls