![Flow diagram, which shows the main steps of the "Tumor-cut" algorithm [1].](https://www.researchgate.net/profile/Stefan-Bauer-6/publication/280046688/figure/fig3/AS:669511061618692@1536635284422/Flow-diagram-which-shows-the-main-steps-of-the-Tumor-cut-algorithm-1_Q320.jpg)
Content uploaded by Stefan Bauer
Author content
All content in this area was uploaded by Stefan Bauer on Oct 04, 2015
Content may be subject to copyright.
Content uploaded by Stefan Bauer
Author content
All content in this area was uploaded by Stefan Bauer on Oct 04, 2015
Content may be subject to copyright.
Proceedings of the MICCAI Challenge on Multimodal
Brain Tumor Image Segmentation (BRATS) 2012
Bjoern Menze, Andras Jakab, Stefan Bauer, Mauricio Reyes, Marcel
Prastawa, Koen Van Leemput
To cite this version:
Bjoern Menze, Andras Jakab, Stefan Bauer, Mauricio Reyes, Marcel Prastawa, et al.. Proceed-
ings of the MICCAI Challenge on Multimodal Brain Tumor Image Segmentation (BRATS)
2012. Bjoern Menze and Andras Jakab and Mauricio Reyes and Stefan Bauer and Marcel
Prastawa and Koen Van Leemput. MICCAI Challenge on Multimodal Brain Tumor Image
Segmentation (BRATS), Oct 2012, Nice, France. MICCAI, pp.77, 2012. <hal-00912935>
HAL Id: hal-00912935
https://hal.inria.fr/hal-00912935
Submitted on 11 Dec 2013
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destin´ee au d´epˆot et `a la diffusion de documents
scientifiques de niveau recherche, publi´es ou non,
´emanant des ´etablissements d’enseignement et de
recherche fran¸cais ou ´etrangers, des laboratoires
publics ou priv´es.
!"!#
! " " #
" #
#
$
!%&#
'()
* *
* ** +,(+# - ( +,(+
./
/ !
All images show low- or high-
grade glioma cases.
0
%
1
$ (+
%
$
-
$
%
' $
'
!
"#$%#
$
' $2%34.!#
52%3467#
&6#
!6#
066#
89:3 &%67#
%&'
0!;63
& ""'!(
""
7567 3&*
)*+,
056 <33 &
" -!!")*+,
**/&
%%
./0/12/"/-/3 ./4/5
3/. !6//*- */.///7777777777!"%
&%
3*!$
/*/+/ /8/'/9/77777!"10
&7/<&
* !
2/1 /4/9/-777777777777777777777 !"%4
%&6=%=
%&7
/4 1/;7777777777777777777777777777777!"%9
%&< *
2
/0-8/8//3777777777777777777777777777!"#4
0< !/&
% !9
9//*/77777777777777777777777777777!"#8
9
3*:!
%&
4/'3/-777777777777777777777777777777777777777!"32
3!; %
&
5/?/477777777777777777777777777777777777!"@6
%&*
<:
?/* '+//87777777777777777777777 !"41
&< 6<
:&
/4// !./-/'/89/-
/17777777777777777777777777777777777777!"B9
&< 6<
7 :&
/4/2/1 9/-1/7777777777777!"=6
%&:
*/((/ !/4/777777777777777!"<4
Context-sensitive Classification Forests
for Segmentat io n of Brain Tumor Tissues
D. Zikic
1
, B. Glocker
1
, E. Konukoglu
1
, J. Shotton
1
, A. Criminisi
1
,
D. H. Ye
2
, C. Demiralp
3
,
O. M. Thomas
4,5
, T. Das
4
, R. Jena
4
, S. J. Price
4,6
1
Microsoft Research Cambridge, UK
2
Department of Radiology, University of Pennsylvania, Philadelphia, PA, US A
3
Brown Universi ty, Providence, RI, USA
4
Cambridge University Ho s p it a ls , Cambridge, UK
5
Department of Radiology, Cambridge University, UK
6
Department of Clinical Neurosciences, Cambridge University, UK
Abstract. We describe our submission to the Brain Tumor Segmenta-
tion Challenge (BraTS) a t MICCAI 2012, which is base d on o u r method
for tissue-specific segmentation of high-grade brain tumors [3].
The main id e a is to cast the segmentation as a classi fi c a ti o n task, and
use the discriminative power of context information. We realize this idea
by equ i p p in g a classification forest (CF) with spatially non-local features
to re p resent th e data, and by providing the CF with initial prob a b il ity
estimates for the single tissue c l a sse s as additional input (along-side the
MRI channels). The initial probabilities are patient-specific, and com-
puted at test time based on a learned model of intensity. Through the
combination of the initial probabilities and the non-local features, our
approach is able to capture t h e context information for each data point.
Our method is fully au t o ma t i c, with segmentation run times in the
range of 1-2 minutes per pati ent. We evaluate the submission by cros s-
validation on the real and s ynthetic, high- and low-grade tumor BraTS
data sets.
1 Introduction
This BraTS submission i s based on our work presented in [3]. We approach the
segmentation of the tumor tissues as a classification problem, where each point
in the br ain is assigned a certain tissue class. The basic building block of our
approach is a standard classi fi cat i on forest (CF), which is a discriminative multi-
class classification method. Classi fi cat i on forests al l ow us to describe brain points
to be cl ass i fie d by very high-dimensional features, which are able to capture
information about the spatial context. These f eat u r es are based on the multi -
channel intensities and are spatially non-local. Furthermore, we augment the
input dat a to the classification forest with initial tissue pr ob abi l i ti e s, which are
estimated as posterior probabilities resulting from a generative intensity-based
model, parametrized by Guassian Mixture models (GMM). Together with the
Proc MICCAI-BRATS 2012
1
2 D. Zikic et al.
(B) initial tissue probabilities (A) input data: MRI
(C) segmentation
Classification Forest
• spatially non-local, context-sensitive features
• simultaneous multi-label classification
Estimation of initial probabilities:
posteriors based on tissue-specific
intensity-based models (GMM-based)
Model-B Model-T Model-E
p
B
p
T
p
E
T1-gad
T1
T2
FLAIR
Fig. 1: Schematic Method Overv i ew : Based on the input data (A), we first
roughly estimate the initial probabilities for the single tissues (B), based on
the local intensity information alone. In a second s t ep , we combine the initial
probabilities (B) with the input data from (A ), resulting in a higher -d i men si on al
multi-channel inp ut for the classification forest. The forest compu t es the segmen-
tation (C) by a simultaneous multi-l abel classification, based on non-local and
context-sensitive feat ur es .
context-sensitive features, the initial p r obab i l it i es as additional input i nc r ease
the amount of context information and thus improve th e classification results.
In this paper, we focus on describing our BraTS submission. For more d et ai l s
on motivation for our approach and relation to previous work, please see [3].
2 Method: Context-sensitive Clas s ifica ti on Forests
An overview of our approach is given in Figure 1. We use a standard classi-
fication forest [1], based on spatially non-local features, and combine it with
initial probability estimates for the individual tissue classes. The initial tissue
probabilities are based on local intensity information alone. They are estimated
with a parametric GMM-based model, as described in Section 2.1. The initial
probabilities are then used as add i t i onal input channels for the fores t, together
with the MR image data I.
In Section 2.2 we give a brief description of classification forest s . The types
of the context-sensitive fe atu r es are described in Section 2.3.
We classify three classes C = {B, T, E} for background (B), tumor (T), and
edema (E). The MR input data is d en ot ed by I = (I
T1C
, I
T1
, I
T2
, I
FLAIR
).
2.1 Estimating Initial Tissue Probabilities
As the first step of our appr oach, we estimate t he initial class probabilities for a
given patient based on the intensity representation in th e MRI input data.
The initial probabilities are computed as posterior probabilities based on the
likelihoods obt ai ne d by training a set of GMMs on t h e training data. For each
Proc MICCAI-BRATS 2012
2
Context-sensitive Classification Forests for Segmentation of Brain Tumors 3
class c ∈ C, we train a single GMM, which captures the likelihood p
lik
(i|c) of the
multi-dimensional intensity i ∈ R
4
for the class c. With the trained likelihood
p
lik
, for a given test patient data set I, the GMM-based posterior prob abi l i ty
p
GMM
(c|p) for the class c is estimated for each point p ∈ R
3
by
p
GMM
(c|p) =
p
lik
(I(p)|c) p(c)
P
c
j
p
lik
(I(p)|c
j
) p(c
j
)
, (1)
with p(c) denoting the prior probabil ity for the class c, computed as a normalized
empirical histogram. We can now use the post er i or probabilities directly as input
for the classification forests, in addition to the multi-channel MR d ata I. So
now, with p
GMM
c
(p):= p
GMM
(c|p), our data for one patient consist s of the following
channels
C = (I
T1-gad
, I
T1
, I
T2
, I
FLAIR
, p
GMM
AC
, p
GMM
NC
, p
GMM
E
, p
GMM
B
) . (2)
For simplicity, we will denote single channels by C
j
.
2.2 Classification Forests
We employ a classification forest ( CF ) to determine a class c ∈ C for a given spa-
tial input point p ∈ Ω from a spatial domain Ω of the patient. Our classification
forest operates on the representation of a spatial point p by a corresponding
feature vector x(p, C), which is based on spatially non-loc al information from
the channels C. CFs are ensembles of (binary) classification trees, indexed and
referred to by t ∈ [1, T ]. As a supervised method, CFs operate in two stages:
training and testing.
During training, each tree t learns a weak class predictor p
t
(c|x(p, C)). The
input training data set is {(x(p, C
(k)
), c
(k)
(p)) : p ∈ Ω
(k)
}, that is, the feature
representations of all spatial points p ∈ Ω
(k)
, in all training patient data sets k,
and the corresponding manual labels c
(k)
(p).
To sim pl i f y notation, we will refer to a data point at p by its feature repre-
sentation x. The set of all data poi nts shall be X.
In a classificat ion tree, each node i contains a set of training examples X
i
,
and a class predictor p
i
t
(c|x), w hi ch is the probability corresponding to t he frac-
tion of points with c l ass c in X
i
(normalized empirical histogram). Starting with
the complete tr ai n in g data set X at the root, the training is perf orm ed by suc-
cessively splitting the training examples at every node based on th ei r feature
representation, and ass i gni n g the partitions X
L
and X
R
to the left and right
child node. At each node, a number of splits along randomly chosen dimensions
of the fe at ur e space is considered, and the one maximizing the Information Gain
is applied (i.e., an axis-aligned hyperplane is used in the split function). Tree
growing is stopped at a certain tree d ep t h D.
At testing, a data point x to be classifi ed is pushed through each tree t, by
applying the learned split functions. Upon arriving at a leaf node l, the leaf prob-
ability is used as the tree probability, i.e. p
t
(c|x) = p
l
t
(c|x). The overall probability
Proc MICCAI-BRATS 2012
3
4 D. Zikic et al.
is computed as the average of tree probabil it i es , i.e. p(c|x) =
1
T
P
T
t=1
p
t
(c|x). The
actual class estimate ˆc is chosen as the class with the highest probability, i.e.
ˆc = arg max
c
p(c|x).
For more details on classification for est s , see for example [1].
2.3 Context-sensitive Feature Types
We employ three features types, which are intensity-based an d parametrized.
Features of these types describe a point to be labeled based on its non-local
neighborhood, such that they are context-sensitive. The first two of these fea-
ture types are quite generic, while the third one is designed with the intuition
of detecting structure changes. We denote the parametrized feature types by
x
type
params
. Each combination of type and parameter settings generates one dimen-
sion in the feature s pac e, that is x
i
= x
type
i
params
i
. Theoretically, the number of
possible combinations of type and parameter settings is infinite, and even with
exhaustive discrete sampling it remains su b st antial. In practice, a certain pre-
defined number d
′
of combinations of f eat ur e ty pes and parameter settings is
randomly drawn for tr ai ni n g. In our experiments, we use d
′
= 2000.
We use the following notation: Again, p is a spatial point, to be assigned a
class, and C
j
is an input channel. R
s
j
(p) denot es an p-centered and axis aligned
3D cuboid r egi on in C
j
with edge len gt hs l = (l
x
, l
y
, l
z
), and u ∈ R
3
is an offset
vector.
– Feature Ty pe 1: measures the intensity difference between p in a channel
C
j
1
and an offset point p + u in a channel C
j
2
x
t1
j
1
,j
2
,u
(p, C) = C
j
1
(p) − C
j
2
(p + u) . (3)
– Feature Typ e 2: measures the difference between int en si ty means of a
cuboid around p in C
j
1
, and around an offset point p + u in C
j
2
x
t2
j
1
,j
2
,l
1
,l
2
,u
(p, C) = µ(R
l
1
j
1
(p)) − µ(R
l
2
j
2
(p + u)) . (4)
– Feature Type 3: capture s the intensity range along a 3D line between
p and p+ u in one channel. This type is designed with the intuition that
structure changes c an yield a large intensity change, e.g. NC being dark and
AC bright in T1-gad.
x
t3
j,u
(p, C) = max
λ
(C
j
(p + λu)) − min
λ
(C
j
(p + λu)) with λ ∈ [0, 1] . (5)
In the exper im ents, the types and parameters are drawn uniformly. The
offsets u
i
originate from t h e range [0, 20]mm, and the cuboid lengths l
i
from
[0, 40]mm.
Proc MICCAI-BRATS 2012
4
Context-sensitive Classification Forests for Segmentation of Brain Tumors 5
Dice score
High-grade (real) Low-grade (real) High-grade (synth) Low-grade (synth)
Edema
Tumor
Edema
Tumor
Edema
Tumor
Edema
Tumor
mean
0.70
0.71
0.44
0.62
0.65
0.90
0.55
0.71
std. dev.
0.09
0.24
0.18
0.27
0.27
0.05
0.23
0.20
median
0.70
0.78
0.44
0.74
0.76
0.92
0.65
0.78
Table 1: Evaluation summary. The Dice scores are computed by the online eval-
uation tool provided by the organizers of the BraTS challenge.
3 Evaluat io n
We evaluate our ap p roach on the re al and synthetic data from the BraTS chal-
lenge. Both real and synthetic exampl es contain separate high-grade (HG) and
low-grade (LG) d at a sets. This results in 4 data sets (Real-HG, Real-LG, Synth-
HG, Synth-LG). For each of these data set s, we perform the evaluat i on inde-
pendently, i.e., we use only the data from one data set for the training and the
testing for this data set.
In terms of sizes, Real-HG contains 20 patients, Synth-LG has 10 patients,
and the two synthetic data sets contain 25 patients each. For the real data sets,
we test our approach on each patient by leave-one-out cross-validation, meaning
that for each patient, the training is performed on all other images from the
data set, excluding the tested im age itself. For th e synthetic images, we perform
a leave-5-out cross-validation.
Pre-processing. We apply bias-field normalization by the ITK N3 implementa-
tion from [2]. Then, we align the mean intensities of the images within each
channel by a global multiplicative factor. For speed reasons, we run the eval-
uation on a down-sampled version of the input images, with is ot rop i c spatial
resolution of 2mm. The computed segmentations are up-sampled back to 1mm
for the evaluation.
Settings. In all tests, we employ forests with T = 40 trees of depth D = 20.
Runtime. Our segmentation method is fully autom at ic , with segmentation run
times in the range of 1-2 minutes per patient. The training of one tree takes
approximately 20 minutes on a singl e desktop PC.
Results. We evaluated our segmentations by the BraTS online evaluati on tool,
and we summarize the results for the Dice score in Table 1.
Overall, the results indicate a higher segmentation quality for the high-grade
tumors than for the low-grade cases, and a better per f orm anc e on the synthetic
data than the real data set.
Proc MICCAI-BRATS 2012
5
6 D. Zikic et al.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.2
0.4
0.6
0.8
1
Dice Score
1 2 4 6 8 11 12 13 14 15
0
0.2
0.4
0.6
0.8
1
Dice Score
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Dice Score
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Dice Score
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.2
0.4
0.6
0.8
1
Specificity
1 2 4 6 8 11 12 13 14 15
0
0.2
0.4
0.6
0.8
1
Specificity
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Specificity
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Specificity
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.2
0.4
0.6
0.8
1
Precision
1 2 4 6 8 11 12 13 14 15
0
0.2
0.4
0.6
0.8
1
Precision
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Precision
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Precision
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.2
0.4
0.6
0.8
1
Recall (=Sensitivity)
1 2 4 6 8 11 12 13 14 15
0
0.2
0.4
0.6
0.8
1
Recall (=Sensitivity)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Recall (=Sensitivity)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.2
0.4
0.6
0.8
1
Recall (=Sensitivity)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
2
4
6
8
10
12
14
16
18
SD (mean) [mm]
1 2 4 6 8 11 12 13 14 15
0
2
4
6
8
10
12
14
16
18
SD (mean) [mm]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
2
4
6
8
10
12
14
16
18
SD (mean) [mm]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
2
4
6
8
10
12
14
16
18
SD (mean) [mm]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
10
20
30
40
50
60
70
80
SD (max) [mm]
(a) Real-HG
1 2 4 6 8 11 12 13 14 15
0
10
20
30
40
50
60
70
80
SD (max) [mm]
(b)Real-LG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
70
80
SD (max) [mm]
(c) Synth-HG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
70
80
SD (max) [mm]
(d) Synth-LG
Fig. 2: Per patient evaluation for the four BraTS data sets (Real-HG, Real-
LG, Synth-HG, Synth-LG). We show the results for edema (blue) and tumor
tissue (red) per patient, and indicate the respective median results with the
horizontal lines. We report the foll owing measures: Dice, Specificity, Precision,
Recall(=Sensitivity), Mean Surface Distance (S D) , and Maximal SD.
Further Evaluation. Furthermore, we reprodu ce most of the BraTS measures
(except Kappa) by our own evaluation in Figure 2. It can be seen in Figure
2, that the Specifici ty is not a very discriminative measure in this appli cat i on.
Therefore, we rat h er eval uat e Precision, which is similar in nature, but does
Proc MICCAI-BRATS 2012
6
Context-sensitive Classification Forests for Segmentation of Brain Tumors 7
not take the backgroun d class into account (TN), and is thus more sensitive to
errors.
In order to obtain a better understanding of the data and the performance
of our method we perform three further measurements.
1. In Figure 3, we measure the volumes of the brain, and t h e edema and tumor
tissues for the individual patients. This is done in order to be able to evalu ate
how target volumes influence the segmentation quality.
2. In Figure 4, we report the results for the b asi c types of classification out-
comes, i.e. true positives (TP), false positives (FP), and false negatives (FN) .
It is interesting to note the correlation of the TP values with the tissue vol-
umes (cf. Fig. 3). Also, it seems th at for edema, the error of our method
consists of more FP esti mat e s (wrongly labeled as edema) than FN estimates
(wrongly not labeled as edema), i.e. it performs an over-segmentation.
3. In Figure 5, we r eport additional meas ur es , which might have an application-
specific relevan ce. We compute the overall Error, i.e. the volume of all mi s-
classified points FN + FP, and the corresponding relati ve version , whi ch
relates the error to the target volume T, i . e. (FN + FP)/T. Also, we com-
pute the absolute and the relative Volume Error |T − (TP + FP)| , and
|T − (TP + FP )| /T, which indicate the potential performance for vol u me tr i c
measurements. The volume error is less sensit i ve than the error measure,
since it does not require an overlap of segme ntations but only that the esti-
mated volume is corre ct (volume error can be expressed as |FN − FP|).
Acknowledgments
S. J. Price is funded by a Clinician Scientist Award from the National Institute
for Health Research (NIHR). O. M. Thomas is a Clinical Lecturer supported by
the NIHR Cambridge Biomedical Research Centre.
References
1. A. Criminisi, J. Shotton, and E. Konukoglu. Decision forests: A unified fra m e-
work for classification, regression, density estimation, manifold learning an d semi -
supervised learning. Foundations and Trends in Computer Graphics and Vision,
7(2-3), 2012.
2. N. Tustison and J. Gee. N4ITK: Nick’s N3 ITK implementation for MRI bias field
correction. The Insight Journal, 20 1 0 .
3. D. Zikic, B. Glocker, E. Konukoglu, A. Criminis i, C. Demiralp, J. Shotton, O. M.
Thomas, T. Das, R. Jena, and Price S. J. Decision forests for tissue-specific segmen-
tation of high-grade gliomas in multi-channel mr. In Proc. Medical Image Computing
and Computer Assisted Intervention, 2012.
Proc MICCAI-BRATS 2012
7
8 D. Zikic et al.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.5
1
1.5
2
Brain Volume [x10
3
cm
3
]
1 2 4 6 8 11 12 13 14 15
0
0.5
1
1.5
2
Brain Volume [x10
3
cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Brain Volume [x10
3
cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Brain Volume [x10
3
cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
20
40
60
80
100
120
140
160
180
Tissue Volumes [cm
3
]
(e) Real-HG
1 2 4 6 8 11 12 13 14 15
0
20
40
60
80
100
120
140
160
180
Tissue Volumes [cm
3
]
(f)Real-LG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
20
40
60
80
100
120
140
160
180
Tissue Volumes [cm
3
]
(g) Synth-HG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
20
40
60
80
100
120
140
160
180
Tissue Volumes [cm
3
]
(h) Synth-LG
Fig. 3: Volume statistics of th e BraTS data sets. We compute the brain volumes
(top row), and the volumes of the edema (blue) and tumor (red) tiss ue s per
patient.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.02
0.04
0.06
0.08
0.1
TP / V
1 2 4 6 8 11 12 13 14 15
0
0.02
0.04
0.06
0.08
0.1
TP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
TP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
TP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.02
0.04
0.06
0.08
0.1
FP / V
1 2 4 6 8 11 12 13 14 15
0
0.02
0.04
0.06
0.08
0.1
FP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
FP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
FP / V
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.02
0.04
0.06
0.08
0.1
FN / V
(a) Real-HG
1 2 4 6 8 11 12 13 14 15
0
0.02
0.04
0.06
0.08
0.1
FN / V
(b) Real-LG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
FN / V
(c) Synth-HG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.02
0.04
0.06
0.08
0.1
FN / V
(d) Synth-LG
Fig. 4: We report the values of tru e pos i ti ves (TP), false positives (FP) , an d
false negatives (FN), for edema (blue), and tumor (red) tissues. To make the
values comparable, we report them as percentage of the patient brain volume
(V). Again, horizontal lines represent median values. It is interesting to note the
correlation of the TP values with the tissue volumes (cf. Fig. 3). Also, it seems
that for edema, the error of our method consist s of more FP estimates (wr ongl y
labeled as edema) than FN estimates (wrongly not labeled as edema), i.e. it
performs an over-segmentation.
Proc MICCAI-BRATS 2012
8
Context-sensitive Classification Forests for Segmentation of Brain Tumors 9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
10
20
30
40
50
60
70
80
90
Error (abs.) [cm
3
]
1 2 4 6 8 11 12 13 14 15
0
10
20
30
40
50
60
70
80
90
Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
70
80
90
Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
70
80
90
Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.5
1
1.5
2
Error (rel.)
1 2 4 6 8 11 12 13 14 15
0
0.5
1
1.5
2
Error (rel.)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Error (rel.)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Error (rel.)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
10
20
30
40
50
60
Volume Error (abs.) [cm
3
]
1 2 4 6 8 11 12 13 14 15
0
10
20
30
40
50
60
Volume Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
Volume Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
10
20
30
40
50
60
Volume Error (abs.) [cm
3
]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 22 24 25 26 27
0
0.5
1
1.5
2
Volume Error (rel.)
(a) Real-HG
1 2 4 6 8 11 12 13 14 15
0
0.5
1
1.5
2
Volume Error (rel.)
(b) Real-LG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Volume Error (rel.)
(c) Synth-HG
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0
0.5
1
1.5
2
Volume Error (rel.)
(d) Synth-LG
Fig. 5: We further evaluate additional measur es which might have app li c ati on -
specific relevance. Again, we have blue=edema, red=tumor, and horizontal
line=median. In the two top rows, we compute the Error, i.e. th e volume of all
misclassified points FN + FP, and the relative version, which relates the error to
the target volume T, i.e. (FN + FP)/T. In the bottom two rows, we compute the
absolute and the relative Volume Error |T−(TP+FP)|, and |T−(TP+FP)|/T.
Proc MICCAI-BRATS 2012
9
Segmentation o f Brain Tumor Image s
Based on Integrated Hierarchical
Classification and Regularization
Stefan Bauer
1
, Thomas Fejes
1,2
, Johannes Slotboom
2
,
Roland Wiest
2
, Lutz-P. Nolte
1
, and Mauricio Reyes
1
1
Institute for Surgical Technology and Biomechanics, University of Bern
2
Inselspital, Bern University Hospital, Switzerland
stefan.bauer@istb.unibe.ch
Abstract. We propose a fully automatic method for brain tumor seg-
mentation, wh ich integrates random forest classific a ti o n with hierarchi-
cal conditional random field regulari za t io n in an energy minimization
scheme. It has been evaluated on the BRATS2012 dataset, which con-
tains low- and high-grade gliomas from simulated and real-patient im-
ages. The method achieved convincing results (average Dice coefficient:
0.73 and 0.59 for tumor and edema respectively) within a reasonably fast
computation t i me (approximately 4 to 12 minutes).
1 Introduction
Fast and accurate segmentation of brain tumor images is an important but
difficult task in many clinical applications. In recent years, a number of different
automatic approaches have be en proposed [1], but desp i t e significant intra- and
inter-rater variabi l i t ie s and the large time consump t i on of manual segmentation,
none of the automatic appr oaches is in routine clinical use yet. However, with
the antici pat e d shift from diameter-based criteria to volume-based criteria in
neuroradiological brain tumor assessment , this is likely to change in the future.
We are presenting a fully automatic me th od for brain tumor segmentation,
which is based on classificati on with integrated hierarchical regularization. Not
only does i t offer to separate healthy from pathologic tissues, but it also subcat-
egorizes the healthy tissues into CSF, WM, GM and the pathologic tiss ue s into
necrotic, active and edema compartment.
2 Methods
The general idea is based on a previous approach presented in [2]. After prepro-
cessing (denoising, bias-field correction, rescaling and histogram matching) [6],
the segmentation task is modeled as an energy mi ni mi zat i on problem in a condi-
tional r and om field (CRF) [8] for mulation. The energy consists of t h e sum of the
singleton potentials in t h e first term and the pairwise potentials in the second
Proc MICCAI-BRATS 2012
10
2
term of equation (1). The expression is minimized using [7] in a hierarchical way
similar to [2].
E =
X
i
V (y
i
, x
i
) +
X
ij
W (y
i
, y
j
, x
i
, x
j
) (1)
The singleton potentials V (y
i
, x
i
) are computed accor d i ng to equation (2),
where y
i
is the label output fr om the classifier, x
i
is the feature vector and δ is
the Kronecker-δ function.
V (y
i
, x
i
) = p(y
i
|x
i
) · (1 − δ(˜y
i
, y
i
)) (2)
In cont ras t to our previous approach, here we make u se of random forests [4], [3]
as a clas si fi er instead of support vector machines (SVM ) . Random forests are en-
sembles of decision trees, which are randomly different. Training on each decision
tree is performed by optimizing the parameters of a split function at every tree
node via maximizing the information gain when splitting the trai ni n g data. For
testing, the feature vector is pushed through each tree, applying a test at each
split node until a leaf node is reached. The label posterior is calculat e d by averag-
ing the posterior s of the leave nodes from all trees p(y
i
|x
i
) = 1/T ·
P
T
t
p
t
(y
i
|x
i
).
Compared to SVMs, random forests have the advantage of being able to natu-
rally handle multi-clas s problems and they provide a probabilistic output instead
of hard label separations [5]. We use the probabi l is t ic output for the weighting
factor p( y
i
|x
i
) in equation (2), in order to control the degree of spatial regular-
ization based on the posterior probab i li ty of each voxel label . A 28-di me ns i onal
feature vector is used for t he classifier, which combines the intensities in each
modality with the first-order textures (mean, variance, skewness, kurtosis, en-
ergy, entropy) com pu t ed from local patches around every voxel in each modality.
We have also developed an improved way to compute the pairwise poten-
tials W (y
i
, y
j
, x
i
, x
j
), which account for the spatial regularization. In equation
(3) w
s
(i, j) is a weighting function, which depends on the voxel sp aci n g in each
dimension. The term (1 − δ(y
i
, y
j
)) penalizes different labels of adjacent voxels,
while the intensity term exp
PCD(x
i
−x
j
)
2·¯x
regulates the degree of smoothing
based on the local intensity variation, where PCD is a pseudo-Chebyshev dis-
tance and ¯x is a gene r ali z ed mean intensity. D
pq
(y
i
, y
j
) allows us to incorporate
prior knowledge by penalizing different tissue adjancencies individually.
W (y
i
, y
j
, x
i
, x
j
) = w
s
(i, j)·(1−δ(y
i
, y
j
))·ex p
PCD(x
i
− x
j
)
2 · ¯x
·D
pq
(y
i
, y
j
) (3)
3 Results
The performanc e of the proposed method has been evaluated on the BRATS2012
dataset
3
using 5-fold cross-validation. The BRATS2012 dat ase t contains skull-
3
http://www2.imm.dtu.dk/projects/BRATS 20 12 /
Proc MICCAI-BRATS 2012
11
3
stripped multimodal MR images (T
1
,T
1contrast
,T
2
, Flair) of 80 low- and high-
grade gliomas from simulations and real patient cases (1mm isotropic resolution).
In order to be compatible with the BRATS ground truth, our “necrotic” and
“active” labels were combined to form the “core” label, the “edema” label was
unmo dified and all other labels were ignored.
Quantitative results for different overlap and surface distance metrics, which
were obtained using the BRATS2012 online evaluation tool, are detailed in table
1 and exemplary image results are shown in figure 1. Computation time for the
segmentation range d from 4 to 12 minutes depending on the size of the dataset.
We also compared the proposed approach to our pre vi ou s method [2] which
used SVMs as a classifier instead of random forests and which had a les s sophis-
ticated regularization. With the new method, the computation time could be
reduced by mor e than a factor of two and the accuracy measured by the Dice
coefficient was also impr oved.
Fig. 1. Exemplary image results shown on one axial slice for a high-grade glioma
patient (first row), a low-grade glioma patient (second row), a simulated high-grade
glioma dataset (third row) and a simulated low-grade glioma dataset (last row). Each
row shows from left to right: T
1
,T
1contrast
,T
2
, Flair image and the label map obtained
from the automatic segmentation (color code: red=CSF, green=GM, blue=WM, yel-
low=necrotic, turquoise=active, pink=edema.)
Proc MICCAI-BRATS 2012
12
4
Table 1. Quantitative results from the BRATS2012 online evaluation tool. HG stands
for high-grade, LG for low-grade and Sim for the simulated glioma d a t as et s. The metrics
in the table from left to right are: Dic e, Jaccard, sensitivity, specificity, average distan c e,
Hausdorff distan c e, Cohen’s kappa.
Dice Jaccard Sens. Spec. AD [mm] HD [mm] Kappa
HG
edema 0.61±0.15 0.45±0.15 1.0±0.0 0.56±0.15 5.0±5.3 60±31
0.32±0.25
tumor 0.62±0.27 0.50±0.25 1.0±0.0 0.59±0.31 6.3±7.8 69±25
LG
edema 0.35±0.18 0.23±0.13 1.0±0.0 0.49±0.23 10.4±9.2 69±28
0.07±0.23
tumor 0.49±0.26 0.36±0.24 1.0±0.0 0.49±0.28 5.4±3.8 53±32
Sim-HG
edema 0.68±0.26 0.56±0.26 1.0±0.0 0.90±0.07 1.3±0.7 12±6
0.67±0.13
tumor 0.90±0.06 0.81±0.09 1.0±0.0 0.91±0.08 1.5±1.7 16±10
Sim-LG
edema 0.57±0.24 0.44±0.22 1.0±0.0 0.84±0.17 1.6±0.9 10±6
0.38±0.18
tumor 0.74±0.10 0.59±0.12 1.0±0.0 0.77±0.18 2.6±1.1 16±5
All
edema 0.59±0.24 0.45±0.23 1.0±0.0 0.75±0.22 3.5±5.1 30±31
0.42±0.27
tumor 0.73±0.22 0.61±0.23 1.0±0.0 0.73±0.26 3.6±4.6 34±29
4 Discussion and Conclusion
We have presented a method for fully automatic segmentation of brain tumors,
which achieves convincing results within a reasonable com pu t at ion time on cl in i -
cal and simulated multimodal MR images. Thanks to the ability of the approach
to delineate subcompartments of healthy and pathologic tissues, it can have a
significant impact in clinical applications, especially tumor volumetry. To evalu-
ate this more thoroughly, a prototype of the method is currently being integrated
into the neuroradiology workflow at Insels pi t al , Bern Univers ity Hospital.
References
1. Angelini, E.D., Clatz, O., Mandonnet, E., Konukoglu, E., Capelle, L., Duffau, H.:
Glioma Dyna mic s and Computational Models: A Review of Segmentation, Reg is -
tration, and I n Silico Growth Algorithms and their Clinical Applications. Current
Medical Imagin g Reviews 3(4) (2007)
2. Bauer, S., Nolte, L.P., Reyes, M.: Fully automatic segmentation of brain tumor
images using support vector machine classification in combination with hierarchi-
cal c o n d it ion a l random field regularization. In: MICCAI. LNCS, vol. 14. Springer,
Toronto (2011)
3. Bochkanov, S., Bystritsky, V.: ALGLIB, www.alglib.net
4. Breiman, L.: Random forests. Machine Learning 45(1) (2001)
5. Criminisi, A., Shotton, J., Konukoglu, E.: Decision Forests for Classification , Re-
gression , Density Estimation , Manifold Learning and Semi-S u perv is ed Learning.
Tech. rep., Microsoft Research (2011)
6. Ibanez, L., Schroeder, W., Ng, L., Cates, J., Others: Th e ITK software guide ( 2 0 0 3 )
7. Komodakis, N., Tziritas, G., Paragios, N.: Performance vs computational efficiency
for optimizing single and dynamic MRFs: Setting the state of the art with primal -
dual strategies. Computer Vision and Imag e Understanding 112(1) (2008)
8. Lafferty, J., McCa ll u m, A., Pereira, F.: Conditional random fields: Probabilistic
models for segmenting and labeling sequence data. In: ICML Proceedings. Citeseer
(2001)
Proc MICCAI-BRATS 2012
13
Spatial Decision Forests for Glioma
Segmentation in Multi-Channel MR Images
E. Geremia
1
, B. H. Menze
1,2
, N. Ayache
1
1
Asclepios Research Project, INRIA Sophia-Antipolis, France.
2
Computer Science and Artificial Intelligence Laboratory, MIT, USA.
Abstract. A fully automatic algorithm is presented for the automatic
segmentation of gliomas in 3D MR images. It builds on the discriminative
random decision forest framework to provide a voxel-wise probabilistic
classification of the volume. Our method uses multi-channel MR intensi-
ties (T1, T1C, T2, Flair), spatial prior and long-range comparisons with
3D regions to discriminate lesions. A symmetry feature is introduced ac-
counting for the fact that gliomas tend to develop in an asymmetric way.
Quantitative evaluation of the data is carried out on publicly available
labeled cases from the BRATS Segmentation Challenge 2012 dataset and
demonstrates improved results over the state of the art.
1 Materials and methods
This section describes our adaptation of the random decision forests to the seg-
mentation of gliomas and illustrates the visual features employed.
1.1 Dataset
To calculate the local image features – b oth during training and for predic-
tions – we performed an intensity normalization [1]. For each data group (i.e.
BRATS
HG and BRATS LG), we fitted the intensity histogram of each sequence
(T1, T1C, T2 and FLAIR) to a reference case. Then image features are calcu-
lated for each voxel v. Features include local multi-channel intensity (T1, T1C,
T2, Flair) as well as long-range displaced box features such as in [2]. In ad-
dition we also incorporate symmetry features, calculated after estimating the
mid-sagittal plane [3]. In total, every voxel is associated with a 412−long vector
of feature responses.
We will adhere to the following notation: the data consists of a collection
of voxel samples v = (x, C), each characterized by a position x = (x, y, z)
and associated with a list of signal channels C. Signal channels C = (I, P)
include multi-sequence MR images I = (I
T 1
, I
T 1C
, I
T 2
, I
F lair
) and spatial priors
P = (P
W M
, P
GM
, P
CSF
). Anatomical images and spatial priors, although having
different semantics, can be treated under the unified term “signal channel”. We
account for noise in MR images by averaging values over a 3
3
voxels box centered
on x, such an average is noted C
c
(x), e.g. C
c
= I
F lair
or P
GM
.
Proc MICCAI-BRATS 2012
14
1.2 Context-rich decision forest
Our detection and segmentation problem can b e formalized as a multi-class
classification of voxel samples into either background, edema or tumor core.
This classification problem is addressed by a supervised method: discriminative
random decision forest, an ensemble learner using decision trees as base learners.
Decision trees are discriminative classifiers which are known to suffer from over-
fitting. A random decision forest [4] achieves better generalization by growing
an ensemble of many independent decision trees on a random subset of the
training data and by randomizing the features made available to each node
during training [5].
Forest training. The forest has T components with t indexing each tree. The
training data consists in a set of labeled voxels T = {v
k
, Y (v
k
)} where the
label Y (v
k
) is given by an exp ert. When asked to classify a new image, the
classifier aims to assign every voxel v in the volume a label y(v). In our case,
y(v) ∈ {0, 1, 2}, 2 for the tumor core, 1 for edema and 0 for background.
During training, all observations v
k
are pushed through each of the trees.
Each internal node applies a binary test [6–9] as follows:
t
τ
low
,τ
up
,θ
(v
k
) =
true, if τ
low
≤ θ(v
k
) < τ
up
false, otherwise
where θ is a function identifying the visual feature extracted at position x
k
.
There are several ways of defining θ, either as a local intensity-based average,
local spatial prior or context-rich cue. These are investigated in more detail in the
next section. The value of the extracted visual feature is thresholded by τ
low
and
τ
up
. The voxel v
k
is then sent to one of the two child nodes based on the outcome
of this test. Training the classifier means selecting the most discriminative binary
test for each node by optimizing over (τ
low
, τ
up
, θ) in order to maximize the
information gain on the input data partition [10], noted T
p
, defined as follows:
IG
τ
low
,τ
up
,θ
(T
p
) = H(T
p
) − H(T
p
|{t
τ
low
,τ
up
,θ
(v
k
)}) where T
p
⊂ T , H stands for
the entropy.
Only a randomly sampled subset Θ of the feature space is available for inter-
nal node optimization, while the threshold space is uniformly discretized. The
optimal (τ
∗
low
, τ
∗
up
, θ
∗
) is selected by exhaustive search jointly over the feature
and threshold space. Random sampling of the features leads to increased inter-
node and inter-tree variability which improves generalization. Nodes are grown
to a maximum depth D. Another stopping criterion is to stop growing a node
when too few training points reach it, i.e. when the information gain is below a
minimal value IG
min
.
As a result of the training process, each leaf node l of every tree t receives a
partition T
l
t
of the training data. The following empirical posterior probability
is then stored at the leaf p
l
t
(Y (v) = b) = |{(v, Y (v)) ∈ T
l
t
|Y (v) = b}|/|T
l
t
|
where b ∈ {0, 1} denotes the background or lesion class, respectively.
Prediction. When applied to a new test data T
test
= {v
k
}, each voxel v
k
is propagated through all the trees by successive application of the relevant
Proc MICCAI-BRATS 2012
15
Fig. 1. 2D view of context-
rich features. (a) A context-rich
feature depicting two regions R
1
and R
2
with constant offset rel-
atively to x. (b-d) Three exam-
ples of randomly sampled features
in an extended neighborhood. (e)
The symmetric feature with respect
to the mid-sagittal plane. (f) The
hard symmetric constraint. (g-i)
The soft symmetry feature consid-
ering neighboring voxels in a sphere
of increasing radius. See text for de-
tails.
binary tests. When reaching the leaf node l
t
in all trees t ∈ [1..T ], posteriors
p
l
t
(Y (v) = c) are gathered in order to compute the final posterior probability
defined as follows: p(y(v) = c) =
1
T
P
T
t=1
p
l
t
(Y (v) = c). The voxel v
k
is affected
the class c ∈ {0, 1, 2} which satisfies c = arg max
c
p(y(v) = c). For each class,
the largest connected component is selected to be the final segmentation.
1.3 Visual features
In this section, two kinds of visual features are computed: 1) local features:
θ
loc
c
(v) = C
c
(x) where c indexes an intensity or a prior channel; 2) context-rich
features comparing the voxel of interest with distant regions . The first context-
rich feature looks for relevant 3D regions R
1
and R
2
to compare within an ex-
tended neighborhood: θ
cont
c
1
,c
2
,R
1
,R
2
(v) = C
c
1
(x) −
1
v ol(R
1
∪R
2
)
P
x
′
∈R
1
∪R
2
C
c
2
(x
′
)
where c
1
and c
2
are two signal channels. The regions R
1
and R
2
are sampled
randomly in a large neighborhood of the voxel v (cf. Fig. 1). The sum over
these regions is efficiently computed using integral volume processing [6]. The
second context-rich feature compares the voxel of interest at x with its symmet-
ric counterpart with respect to the mid-sagittal plane, noted S(x): θ
sy m
c
(v) =
C
c
(x) − C
c
◦ S(x) where c is an intensity channel. Instead of comparing with the
exact symmetric S(x) of the voxel, we consider, respectively, its 6, 26 and 32
neighbors in a sphere S (cf. Fig. 1), centered on S(x). We obtain a softer version
of the symmetric feature which reads: θ
sy m
c,S
(v) = min
x
′
∈S
{C
c
(x) − C
c
(x
′
)}.
2 Results
In our experiments, forest parameters are fixed to the following values: number of
random regions per node |Θ| ≃ 100, number of trees T = 30, tree depth D = 20,
lower bound for the information gain IG
min
= 10
−5
. These values were chosen
based on prior parameter optimization on synthetic data (SimBRATS
HG and
SimBRATS
LG) and worked well for real data too.
Proc MICCAI-BRATS 2012
16
Table 1. Segmentation of high grade gliomas in the BRATS dataset. Dice,
TPR and PPV are reported for the segmentation of the edema only, the core only and
the whole tumor.
Edema Core Tumor
Patient Dice TPR PPV Dice TPR PPV Dice TPR PPV
HG01 0.46 0.72 0.34 0.74 0.77 0.71 0.65 0.84 0.53
HG02 0.58 0.97 0.41 0.65 0.51 0.89 0.61 0.93 0.46
HG03 0.70 0.88 0.58 0.79 0.99 0.65 0.76 0.95 0.63
HG04 0.43 0.69 0.31 0.45 0.36 0.59 0.78 0.91 0.69
HG05 0.49 0.60 0.41 0.39 0.25 0.92 0.54 0.49 0.61
HG06 0.61 0.77 0.51 0.75 0.69 0.82 0.75 0.84 0.68
HG07 0.63 0.68 0.58 0.76 0.63 0.96 0.70 0.70 0.70
HG08 0.73 0.78 0.69 0.63 0.65 0.62 0.84 0.89 0.80
HG09 0.80 0.81 0.77 0.69 0.55 0.93 0.84 0.79 0.90
HG10 0.00 0.00 0.00 0.80 0.69 0.96 0.09 0.20 0.05
HG11 0.69 0.78 0.61 0.81 0.87 0.76 0.83 0.92 0.75
HG12 0.67 0.88 0.54 0.00 0.00 0.00 0.86 0.91 0.81
HG13 0.49 0.85 0.35 0.92 0.98 0.87 0.66 0.96 0.51
HG14 0.33 0.81 0.20 0.47 0.31 0.92 0.84 0.84 0.84
HG15 0.67 0.83 0.57 0.83 0.76 0.91 0.78 0.86 0.71
HG22 0.63 0.90 0.49 0.51 0.36 0.86 0.69 0.77 0.62
HG24 0.52 0.83 0.37 0.67 0.53 0.91 0.57 0.74 0.47
HG25 0.51 0.57 0.46 0.05 0.02 0.95 0.55 0.48 0.64
HG26 0.66 0.57 0.80 0.03 0.02 0.07 0.57 0.45 0.77
HG27 0.57 0.93 0.41 0.57 0.41 0.98 0.74 0.85 0.65
mean 0.56 0.74 0.47 0.58 0.52 0.76 0.68 0.77 0.64
std 0.17 0.21 0.19 0.27 0.30 0.28 0.18 0.20 0.18
For quantitative evaluation, a three-fold cross-validation is carried out on
this dataset: the forest is trained on
2
3
of the cases and tested on the other
1
3
,
this operation is repeated three times in order to collect test errors for each case.
Note that the random forest is trained on the preprocessed data. Prediction on
a single image lasts for approximately 10 minutes.
The binary classification is evaluated using two measures, true positive rate
(TPR) and positive predictive value (PPV), both equal 1 for perfect segmen-
tation. Formally, Dice =
T P
F P +2·T P +F N
, T P R =
T P
T P +F N
and P P V =
T P
T P +F P
where T P counts the number of true positive voxels in the classification com-
pared to the ground truth, F P the false positives, F N the false negatives.
Aknowledgments
This work was partially supported by the European Research Council through
the ERC Advance Grant MedYMA on Biophysical Modelling and Analysis of
Dynamic Medical Images.
Proc MICCAI-BRATS 2012
17
Table 2. Segmentation of low grade gliomas in the BRATS dataset. Dice,
TPR and PPV are reported for the segmentation of the edema only, the core only and
the whole tumor.
Edema Core Tumor
Patient Dice TPR PPV Dice TPR PPV Dice TPR PPV
LG01 0.00 0.00 0.00 0.83 0.92 0.76 0.71 0.67 0.77
LG02 0.43 0.35 0.56 0.32 0.23 0.49 0.70 0.55 0.96
LG04 0.46 0.35 0.66 0.05 0.16 0.03 0.62 0.62 0.62
LG06 0.45 0.41 0.48 0.18 0.99 0.10 0.49 0.87 0.34
LG08 0.30 0.29 0.32 0.44 0.37 0.55 0.71 0.63 0.81
LG11 0.21 0.46 0.13 0.14 0.24 0.10 0.47 0.86 0.32
LG12 0.26 0.52 0.17 0.00 0.00 0.00 0.49 0.62 0.40
LG13 0.22 0.27 0.18 0.00 0.00 0.00 0.42 0.32 0.61
LG14 0.19 0.20 0.19 0.00 0.00 0.00 0.34 0.47 0.27
LG15 0.34 0.34 0.34 0.00 0.00 0.00 0.22 0.29 0.18
mean 0.29 0.32 0.30 0.20 0.29 0.20 0.52 0.59 0.53
std 0.14 0.15 0.21 0.27 0.37 0.28 0.17 0.19 0.26
References
1. Coltuc, D., Bolon, P., Chassery, J.M.: Exact histogram specification. IEEE TIP
15 (2006) 1143 –1152
2. Geremia, E., Clatz, O., Menze, B.H., Konukoglu, E., Criminisi, A., Ayache, N.:
Spatial decision forests for ms lesion segmentation in multi-channel magnetic res-
onance images. NeuroImage 57 (2011) 378–90
3. Prima, S., Ourselin, S., Ayache, N.: Computation of the mid-sagittal plane in 3d
brain images. IEEE Trans. Med. Imaging 21(2) (2002) 122–138
4. Amit, Y., Geman, D.: Shape quantization and recognition with randomized trees.
Neural Computation 9(7) (1997) 1545–1588
5. Breiman, L.: Random forests. Machine Learning 45(1) (2001) 5–32
6. Shotton, J., Winn, J.M., Rother, C., Criminisi, A.: Textonboost for image un-
derstanding: Multi-class object recognition and segmentation by jointly modeling
texture, layout, and context. Int. J. Comp. Vision 81(1) (2009) 2–23
7. Yi, Z., Criminisi, A., Shotton, J., Blake, A.: Discriminative, semantic segmentation
of brain tissue in MR images. LNCS 5762, Springer (2009) 558–565
8. Lempitsky, V.S., Verhoek, M., Noble, J.A., Blake, A.: Random forest classification
for automatic delineation of myocardium in real-time 3D echocardiography. In:
FIMH. LNCS 5528, Springer (2009) 447–456
9. Criminisi, A., Shotton, J., Bucciarelli, S.: Decision forests with long-range spatial
context for organ localization in CT volumes. In: MICCAI workshop on Proba-
bilistic Models for Medical Image Analysis (MICCAI-PMMIA). (2009)
10. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)
Proc MICCAI-BRATS 2012
18
Multimodal Brain Tumor Segmentation Using The
“Tumor-cut” Method on The BraTS Dataset
Andac Hamamci, Gozde Unal
Faculty of Engineering and Natural Sciences, Sab a n c i University, Istanbul, Turkey
gozdeunal@sabanciuniv.edu
Abstract. In this paper, the tumor segment a t io n method used is described
and the experimental results obtained are reported for the “BraTS 2012 - Mul-
timodal Brain Tumor Seg m entation Challenge” of MICCA I ’1 2 . “Tumor-cut”
method, presented in [1 ] is adapted to multi-modal data to include edema seg-
mentation. The method is semi-automatic, requiring the user to draw the max-
imum diameter of the tumor, which ta kes about a minute user-interaction time
per case. The typi c a l run-ti me for each c a se is around 10-20 minutes dep en d i n g
on the size o f the tumor. Overall Dice overlap with the expert segmentation is
0.36 ± 0.25 for the edema/infiltration and 0.69 ± 0.20 for t h e tumor region.
1 Introduction
In our “Tumor-cut” paper, the semi-supervise d tumor segmentation method is de-
scribed, in detail [1]. This method, specifical l y targets gross tumor volume (GTV) of
the brain tumors on the contrast enhanced T1-weighted MR images (T1C). Here, we
extend our method to multi-modal MRI (T1C + FLAIR) to include edema segmen-
tation, and also evaluated on low-grade non- en han ce d tumors. Although , we segment
the necrotic areas of the high-grade tumors, quantitative necrotic segmentation result s
are not reported in this paper due to the lack of the groun d truth labels of necrotic
regions.
2 Methods
The main steps of the “Tumor-cut” method, presented in [1], is given in Fig. 1. The
weight of the re gul ar iz er of the level-set is set to 0.2, which is determined in [1] by
experiments on both real and synthetic dat a. The same algorithm is applied on FLAIR
volumes to segment the clinical tumor volume (CT V = GT V + Edema). This ti me,
the user is asked to draw the maximum diameter of the edema region visible on FLAIR
images. The main differences observed between GTV (on T1C) and CTV (on FLAIR)
is that the sphericity of th e ed ema area is lower and there occurs more curvatures.
For the FLAIR segmentation, to allow results w it h more curvatures, a four times
lower regularizer weight is set arbitrarily, at 0.05. The user-input is gath er ed by two
different maximum diameter lines, drawn separately, one for FLAIR and one for T1C
volumes. The resulting maps are combined simply by assigning tumor labels using
T1C segmentation and assigning edema label to the difference area of the FLAIR
segmentation minus the T1C segmentation.
V
Edema
= {x ∈ V
F LAIR
|x /∈ V
T 1C
}
Proc MICCAI-BRATS 2012
19
Fig. 1: Flow diagram, which shows the main steps of the ”Tumor-cut” algorith m [1].
Table 1: Overall results obtained on the BraTS dataset.
Dice Overlap Jaccard Score Specificity Sensitivity Cohen’s
Edema / Tumor Edema / Tumor Edema / Tumor Edema / Tumor Kappa
0.37 ± 0.25/0.69 ± 0.19 0.25 ± 0.20/0.56 ± 0.23 0.99 ± 0.01/1.00 ± 0.00 0.54 ± 0.33/0.86 ± 0.19 0.27 ± 0.23
3 Results
The method is implement ed on Matlab environment, running on a windows 7 worksta-
tion, using mex files for core algorithms. For each case, user interaction takes about a
minute and typ ic al run-time for each case is around 10-20 minutes, depending on the
size of t he tumor. The dat as et is downloaded from Kitwar e/M i das web site and th e
online system provided by the “Virtual Skeleton Database” is used for the evaluation.
The Dice overlap results of the runs on each case of the BraTS data set is tabulated in
Table 2. For each subset, including high grade and low grade, si mulated and patient
data, the Dice overlap scores obtained are given as bar charts in Figs 3-6. The overall
Dice overlap, Jaccard scores, Sensitivity/Specificity and Cohen’s Kappa results wi t h
the standard deviations ar e reported in Table 1.
4 Discussion and Conclusions
Originally we limited the scope of the “Tumor-cut” algorithm to the contrast enhanced
gross tumor volumes, which corresponds to the active parts of the high grade tumors
in Table 2. For the high-grade cases, the results obtained on the patient dataset (0.73)
Proc MICCAI-BRATS 2012
20
Fig. 2: Table of the Dice overlap re su l t s obtained on each case.
Fig. 3: Dice overlap results obtained on each case of the low-grade patient subset.
and the simulated dataset (0.80) are consist ent with the results reported in [1] (0.80
and 0.83). Because the edema region is calculated by substraction of the two maps,
the overlap scores for edema is not independent of the scores for the tumor. For t he
low-grade simulated data, low performance in some cases is m ost l y due to the low
number of voxels labeled as edema -comparing to patient cases- in the groun d truth
segmentation, which causes low overlap scores .
Acknowledgement. This work was partially supported by TUBA-GEBIP (Turkish
Academy of Sciences) and EU FP7 Grant No: PIRG03-GA-2008-231052.
Proc MICCAI-BRATS 2012
21
Fig. 4: Dice overlap results obtained on each case of the high-grade patient subset.
Fig. 5: Dice overlap results obtained on each case of the simulated low-grade subset.
References
1. Hamamci, A., Kucuk, N., Karaman, K., E n g in , K., Unal, G.: Tumor-cut: Segmentation
of brain tumors on contrast enhanced mr images for radiosurgery applications. Medical
Imaging, IEEE Transaction s on 31(3) (march 2012) 790 –804
Proc MICCAI-BRATS 2012
22
Fig. 6: Dice overlap results obtained on each case of the simulated high-grade subset.
Proc MICCAI-BRATS 2012
23
Brain tumor segmentation based on GMM and active
contour method with a model-aware edge map
Liang Zhao
1
, Wei Wu
1,2
, Jason J.Corso
1
1 Department of Computer Science and Engineering,Unveristy at
buffalo SUNY at Buffalo,NY,14260,USA
2 School of Information Engineering, Wuhan University of
Technology, Wuhan 430070, China
(lzhao6@buffalo.edu) (wwu25@buffalo.edu) (jcorso@buffalo.edu)
We present a method for automatic segmentation of heterogeneous brain tumors.
Our method will take about 30 minutes to process one volume in Matlab.
1 Theory background
Our method combines the model of gray distribution of pixels (Gaussian Mixture
Models, GMM) with the edge information between two difference classes of tissue in
the brain. High detection precision can be achieved. The core of our method is based
on the following three models.
1.1 Gaussian Mixture Model(GMM)
We model five classes of data in brain: brain white matter, brain gray matter, brain
csf, tumor and edema.
Denote the parameters of Gaussian component [1]!φ
i
= {ϕ
i
, μ
i
, Σ
i
}, where μ
!
is
mean of vector and Σ
!
is the covariance matrix. The ϕ
!
parameter is called the mixing
coefficient and describes the relative weight of component i in the complete model.
The complete model can be written !ψ = {k, φ
!
, ⋯ , φ
!
}, where k is the number of
component in the data. A mixture model on d-dimension data x is written:
P
x; Ψ = ϕ
!
!
!!!
p
x; μ
!
, Σ
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=
ϕ
!
!"#!(!
!
!
!!!
!
T
!
!
!!
!!!
!
)
(!")
!
!
|!
!
|
!
!
!
!!!
(1)
The standard expectation-maximization algorithm is used to estimate the parame-
ters of each class mixture model in a maximum likelihood formulation.
Proc MICCAI-BRATS 2012
24
1.2 Probability distance model between two different tissues
We define a likelihood function P(
s
!
| m
u
) for the probability of the observed
statistics
s
!
conditioned on model variables m
!
of pixels u . Denote b
!
= white
matter, b
!
=gray matter,!b
!
=csf, b
!
=edema,!b
5
=tumor, According to the characteristic
brain MRI images , we assume
p =
m
!
s
!
, s
!
− s
1
, m
!
= p(m
!
|s
!
− s
!
, m
!
) ! (2)
To characterize an edge between normal brain tissues and abnormal brain tissues
(edema) we can deduce the following term (the edge between edema and tumor can
be derived similarly).
P
m
!
= normal!brain, m
!
= abnormal!brain s
!
, s
!
=
p(
!
!!!
!
!!!
m
!
= b
!
, m
!
= b
!
s
!
, s
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=
p(
!
!!!
!
!!!
m
!
= b
!
s
!
, s
!
− s
!
, m
!
= b
!,
p(m
!
= b
!
s
!
=
( p(
!
!!!
!
!!!
m
!
= b
!
s
!
− s
!
, m
!
= b
!,
)
p
s
!
m
!
= b
!
p m
!
= b
!
p(s
!
!
!!!
m
!
= b
!
p m
!
= b
!
=
p s
!
− s
!
|m
!
= b
!
, m
!
= b
!
p(m
!
= b
!
|m
!
= b
!
)
!
!!!
p(s
!!
!
!!!
s
1
|m
!
= b
!
, m
!
= b
!
)p(m
!
= b
!
)
!
!!!
p s
!
m
!
= b
!
p m
!
= b
!
p(s
!
!
!!!
m
!
= b
!
p m
!
= b
!
(3)
1.3 Active contour model
The traditional deformable active contour model [2] is a curve X(S) = [x(s),y(s)],s
∈[0,1], that move within the image to minimize an energy function. The curve dy-
namically changes the shape of an initial contour in response to internal and external
forces. The internal forces provide the smoothness of the contour. While the external
forces push the curve more toward the desired features, such as boundaries. The ob-
ject contour is extracted when the energy function is minimized. The energy is de-
fined as
E =
!
!
!
!
α x
!
s
!
+ β x
"
s
!
+ E
!"#
x
s ds (4)
where,
!x
′
s and x
"
s are first and second derivatives of x s with respect to s. The
parameter α controls the tension of the curve and β controls its rigidity. E
ext
is the
Proc MICCAI-BRATS 2012
25
external energy which is calculated from the image data. To minimize the energy
function, the snake must satisfy the Euler equation:
αx
!!
s − βx
!!!!
s − ∇E
!"#
=0 (5)
According gradient vector flow snake was proposed by Xu and Prince [3], we de-
fine a new static external force field called GVF field:
F
!"#
= v
x, y = [u x, y , v x, y ] (6)
where u and v are the grey changes on x-axis and y-axis of the image
ly.!F
!"#
!!can be computed by minimizing the following energy function:
E =
μ u
!
!
+ u
!
!
+ v
!
!
+ v
!
!
+ |∇f|
!
|v − ∇f|
!
dxdy
where, u
!
, u
!,
v
!
, v
!
!are derivative of x-axis and y-axis respectively. f(x,y) is the edge
map (using probability distance model between two different tissues). µ is a regulari-
zation parameter governing the tradeoff between the first term and the second term in
the formula [4].
2 Algorithm implementation
2.1 Acquiring rough localization information of tumor and edema
On the training data, we run Expectation Maximization for the GMM on each class
of data with free means, variances, and mixing proportions. These means are labeled
and saved for the testing data.
For the testing data, firstly, we use Graph-based Segmentation to get some super-
voxels in the volumes [5]. Secondly, every voxel can be classified by EM for GMM
with fixed means, and labeled according to the labeled means. Thirdly, every super-
pixel can be classified by using maxing vote’s method. Finally, we chose some spe-
cial super-pixels whose 50%
pixels are almost tumor or edema.
2.2 Seeking edges of between different tissues based on probability distance
model
We compute the probability distance for every pixel according to (3). A model-
aware edge map can be got. The pixel-pair-class likelihoods, p
s
!
− s
!
|m
!
=
b
!
, m
!
= b
!
are computed against GMM.
2.3 Acquiring high precision boundary based on snake method
Based on the rough location information achieved in step (1), we can get an initial
contour of the object. And combining with the edges from step (2), precise boundaries
Proc MICCAI-BRATS 2012
26
between tumor and edema, and edema and normal brain tissues are located using
active contour method.
3 Experiment results
We only process the true data, the whole result can be found on the web. We pick
some data and list them as table 1.The figures are raw data, segmentation result with
GMM with fixed means and segmentation result with model-aware Snake.
Table 1. Segmentation performance on part of the training data
Subjcet
Average
Dist 1
Average
Dist 2
Dice
1
Dice
2
Hausdroff
Dist 1
Hausdroff
Dist 2
Cohen’s
Kappa
Sensitivity
1
Sensitivity
2
Specificity
1
averages
17.573
0
0.348
0.307
131.772
114.957
0.156
0.367
0.327
0.999
0.999
BRATS_LG0008
0
0
0.199
0.83
193.891
180.73
0.16
0.281
0.935
0.998
0.999
BRATS_LG0002
123.01
0
0.602
0.322
134.994
143.156
0.353
0.51
0.246
0.997
0.997
BRATS_HG0015
0
0
0.716
0.875
0
141.032
0.623
0.572
0.81
1
1
BRATS_HG0008
0
0
0.728
0.823
113.214
125.41
0.602
0.601
0.751
0.999
0.999
BRATS_HG0006
0
0
0.329
0.786
176.529
0
0.348
0.208
0.81
0.999
0.998
BRATS_HG0003
0
0
0.699
0.912
146.58
130.088
0.554
0.544
0.92
1
1
BRATS_HG0002
0
0
0.834
0.537
157.197
84.94
0.341
0.93
0.756
0.998
0.998
4 Reference
1. Jason J. Corso, Eitan Sharon, Shishir Dube, Usha Sinha and Alan Yuille, “Efficient multi-
level brain tumor segmentation with integrated Bayesian model classification”, IEEE trans.
Med,Imag.,vol.27, no.2, pp.629-640, 2008.
2. Yezzi. A., Kichenassamy.S, Kumar.A, Olver.P, and Tannenbaum.A, “A geometric snake
model for segmentation of medical imagery”, IEEE trans.Med,Imag.,vol.16,no.2,pp.199-
209,1997.
3. C. Xu and J.L. Prince, “Snakes, shapes, and gradient vector flow”, IEEE Trans. on Image
Processing, vol.7,no.3, pp. 359-369, 1998.
4. A. Rajendran, R. Dhanasekaran, “Brain tumor segmentation on MRI brain images with
fuzzy clustering and GVF snake model”, INT J COMPUT COMMUN, Vol.7, No. 3, pp.
530-539, 2012.
5. Xu, C. and Corso, J. J.. “Evaluation of Supervoxel Methods for Early Video Processing”
CVPR 2012.
Proc MICCAI-BRATS 2012
27
Probabilistic Gabor and Markov Random Fields
Segmentation of Brain Tumours in MRI Volumes
N. K. Subbanna and T. Arb el
Centre for Intelligent Machines, McGill University, Montreal, Quebec, Cana d a
Abstract. In this paper, we present a fully automated technique two
stage technique for segmenting brain tumours from multispectral human
brain magnetic resonance images (MRIs). From the training volumes,
we model the brain tumour, o ed ema and the other healthy brain tissues
using their combined space characteristics. Our segmentation technique
works on a c o mbination of Bayesian classification of the Gabor decom-
position of the brain MRI volumes to produce an initial classification
of brain tumours, along with the other classes. We follow our initial
classification with a Ma r kov Random Field (MRF) classification of the
Bayesian output to resolve local inhomogeneities, and impose a smooth-
ing constraint. Our results show a Dice similarity coefficient of 0.668 for
the brain tumours and 0.56 for the oedema.
1 Introduction
Brain tumours are a serious health problem, and it is estimated that roughly
100,000 people are diagnosed with br ain tumours every years. One of the pri-
mary diagnostic and treatment evaluation tools for brain tumours is the mag-
netic resonance image (MRI) of the brain. A reliable method for segmenting
brain tumours would be very useful. However, brain tumours, owing to their ex-
treme diversity of shape, size, type of tumour, etc., pres ent a serious challenge to
segmentation techniques. Given the importance of the problem, over the years,
there have been a large number of techniques attempted to segment brain tu-
mours automatically. Some of the more important techniques include multilevel
segmentation by Bayesian weighted aggregation [1], knowledge based fuzzy tech-
niques [2], and atlas based classification [3]. Wavelet based decompositions are
attractive since they are good at capturing large textur es of the kind found in
brain tumours effectively, and it is unsurprising that there are a few attempts
to employ wavelets. One of the more prominent is the wavelet decomposition
used in conjunction with suppor t vector machines [4]. In this paper, however,
we build on this technique by constructing models not for just the tumours and
the oedema, but also for the healthy tis s u es . We, then, utilise the natural ability
of the combined space featu r es to capture the existing patterns to train the ma-
chine to recognise the patterns of the tumours, and distinguish it from the other
healthy tissu es , and provide us with an initial classification. From this initial
classification, we then use Mar kov Random Fields (MRFs) to capture the local
label homogeneities and also eliminate false positives that occur due to spurious
tumour textures that may arise in other parts of the brain .
Proc MICCAI-BRATS 2012
28
2 Gabor Bayesian Classification
2.1 Training
Our goal is to correctly identify the oedema and the active tumour. During the
initial training ph as e, we first register a tissue atlas to the training volumes ob-
tain the healthy tissues, Grey Matter (GM), White Matter (WM), and Cerebro-
Spinal fluid (CSF). We super impose the tumou r and oedema maps provided by
the experts to obtain all the classes in the training volumes.
We decompose the training volumes into their constituent Gabor filter b an k
outputs. The input volumes are the MRI intensity volumes in the four mod alities ,
viz, T1, T2, T1c and FLAIR, so at each voxel, we have a four dimension al vector
I
i
= (I
T 1
i
, I
T 1c
i
, I
F LAIR
i
, I
T 2
i
). Each image is decomposed to its filter bank output
using multiwindow Gabor trans for ms of the form suggested by [5]. The filter bank
outputs are obtained by convolving each modality volume with the Gabor filter
bank, which is ob tain ed using the equation
h(x, y) =
1
2πσ
x
σ
y
exp
−
1
2
x
2
σ
x
2
+
y
2
σ
y
2
cos(2πu
0
x) (1)
where σ
x
and σ
y
are the spreads in the x and y directions and u
0
is the modu-
lating frequency. In our case, we choose 4 orientations between 0 and π radians
and 5 frequencies. Each chos en frequency is an octave of the previous to ensure
that the entire spectrum is covered. We model each class as a Gaussian mixture
model, and an 8 component Gaussian mixture suffices to model the different
classes in the combined space. We model the Gabor coefficients of all classes,
including the tumour an d the oed ema, using Gaussian mixture models.
2.2 Classification
Once the test volume is obtained, it is decomposed into its Gabor filter bank
outputs using eqn. (1). The class of each voxel is obtained using Bayesian clas-
sification, which is given by
P (C
i
| I
G
i
) ∝ P (I
g
i
| C
i
)P (C
i
), (2)
where C is a random variable that can take the value of the 5 classes, an d
I
G
i
= I
0
i
, I
1
i
, . . . , I
R−1
i
is the s et of R Gabor co efficients of the particular voxel
i. It is our experience that the active tumours are quite correctly determined
by the Gabor Bayesian technique, b u t there are often false positive oedema
segmentations in regions that mimic the presence of oedema.
3 Markov Random Field Classification
The first Bayesian classification results in tumour candidates. We refine this
classification by building an MRF based model. We focus on both the intens ities
Proc MICCAI-BRATS 2012
29
of the voxels and th e intensity differences as contrasts are much more consistent.
The MRF model is based on the intensity of the voxel, th e spatial intensity
differences and the class of n eighbouring voxels. This can be wr itten as
P (C
i
| I
i
, I
N
i
) = P (I
i
| C
i
)P (C
i
)
M−1
X
C
N
i
=0
P (∆I
N
i
| C
i
, C
N
i
)P (C
N
i
| C
i
) (3)
where C
N
i
are the classes of the neighbours of i, ∆I
N−I
= I
N
i
− I
i
.
3.1 Training
Here, we build th e intensity distributions of the classes and model them using
multivariate Gaussian models. For the neighbourhood, we consider an 8 neigh-
bourhood around the voxel in axial plane and the corresponding voxels in the
slices above and b elow, and build the distributions of each pair , trip let an d
quadriplet of classes that have an edge or vertex in common in the defined
neighbourhood using multivariate Gaussian models. This allows us to model
all neighbourhood relations completely in both a mathematical and a practi-
cal sense. We use the initial Gabor classification as the prior with the oedema
probabilities falling sharply away from th e tumour for the second phase.
3.2 MRF Classification
We need to compute P (C | I) where C is a configuration of the labels of all the
vox els in the volume and I is the set of intensities across all the modalities for
all the voxels in the configuration. A sound method of computing P (C | I) is
by considering the problem as an MRF, which suggests that all class labels are
dependent only on their local neighbourhood. Using eqn. (3), we can obtain the
energy funcion for the configuration of labels in the volume with
U(C) =
Z−1
X
i=0
(I
i
− µ
C
i
)
T
Σ
−1
C
i
(I
i
− µ
C
i
)
+
X
N
i
(∆I
N
i
− µ
C
N
i
,C
i
)
T
Σ
−1
C
N
i
,C
i
(∆I
N
i
− µ
C
N
i
,C
i
) + αm(C
N
i
, C
i
), (4)
where ∆I
N
i
= I
N
i
− I
i
, m(C
N
i
, C
i
) = 1 if C
N
i
= C
i
, Z is the total number of
vox els , and 0 otherwise, and α is the weighting coefficient vector. To maximise
P (C), we use iterated conditional modes (ICM) [6] to minimise U (C), where
C
min
= argmin
C∈F
U(C), and F is the set of all possible label con fi gu r ation s .
4 Results
In Fig. 1, we compare the results of the two slices where our results are compared
against th os e of the exp er ts ’ segmentation. In both cases, it can be seen that our
results are comparable to the exp er ts ’ comparison.
Proc MICCAI-BRATS 2012
30
Fig. 1. (a) (b) (c) (d)
(a) Expert labelling of slice 81 of the volume LG0015 and (b) its corresponding algo-
rithmic labelling. (c) Similarly, the expert labelling of slice 93 of the volume HG0015
and (d) its corresponding labelling by the algorithm. As may be seen, visually, our
algorithm’s performance is very close to the experts’ evaluation.
Quantitatively, we train our algorithm on 29 volumes given and test it on the
remaining one in a leave one out fashion. We get a Dice similarity coefficient of
0.561 ± 0.118 for the oedema and 0.668 ± 0.126 for the active tumour when we
compare our segmentation again s t those of experts.
References
1. J. J. Corso, et. al., “Efficient Multilevel Brain Tumour Segmentatin with Integrated
Bayesian Model Classification ” , IEEE Trans. Med. I ma g ., Vol. 27(5), pp. 62 9 -6 4 0 .
2. M. C. Cla r k et. al., “Automatic Tumour Segmentation using Knowledge Based
Techniques”, IEEE Trans. Med. Imag., Vol. 17(2), pp. 187-201.
3. M. Kaus et. al., “Adaptive Template Moderated Brain Tumour Segmentation in
MRI”, Bildverarbeitung fur die Medizin, pp. 102-106.
4. G. Farias et. al., “Brain Tumour Diagn o s is with Wavelets and Support Vector
Machines”, in Proc. 3rd Int. Conf. Intell. Systems and Knowledge Engg., 2008.
5. A. K. Jain, and F. Farrokh n ia , “Unsupervised Texture Segmentation Using Gabor
filters” Pattern Recognition, Vol. 24(12), pp. 1 1 6 7 -1 1 8 6 , 1991.
6. R. O. Duda, P. E. Hart, and D. G. Stork, “Pa tter n Classification”, John Wiley
and Sons, 2000.
Proc MICCAI-BRATS 2012
31
Hybrid Clustering and Logistic Regression
for Multi-Modal Brain Tumor Segmentation
Hoo-Chang Shin
CRUK and EPSRC Cancer Imaging Centre
Institute of Cancer Research and Royal Marsden NHS Foundation Trust
Sutton, United Kingdom
{hoo.shin}@icr.ac.uk
Abstract. Tumor is an abnormal tissue type, therefore it is hard to be
identified by some classical classification methods. It was tried to find
a non-linear decision boundary to classify tumor and edema by a joint
approach of hybrid clustering and logistic regression .
Keywords: Sparse dictiona ry learning, K-means clustering, Logistic Re-
gression
1 Introduction
Classifying tumor is challenging as it represents a coll e ct i on of some abnormal
tissue types which results in not enough labe le d training dataset to “learn”
about the different characteristics of an unseen tumor. In this situation, cluster-
ing would be a viable approach, as it divides a given dataset into a number of
sub-groups without requiring the labeled training dataset. While the mor e clas-
sical clustering method s such as k -me ans and Expectation Maximization (EM)
produce good clustering result, they just divide a given dataset into a number
of sub-group s, such that there is no “learning” process involved that a learned
knowledge can be applied to an unseen d at ase t .
Sparse dictionary learni n g [1, 2] has been applied to a number of wide range
of dis c ip li ne s such as signal reconstruction for me di cal image acquisition [3],
image denoising [4], object recognition [5], and medical image segmentation [6].
By representing a given dataset by a combination of some learned dict i on ar y’ s
basis vector set s, the dataset can be cl u st er ed . This approach, often referred as
sparse c od i ng, was applied for object recognition [7, 8] and multi-modal medical
image segmentation [9].
It was noticed that edema is already quite well classified by this method,
whereas tumor is not. There are different types of tumor in the dataset, t he re for e
there is no clear pattern in the images of different modali ti e s for tumor. Logistic
regression was applied to find a non-linear decision boundary to classify these
different types of tumors from a more normal tissues.This is combined with
volume-wise k-me ans clusteri ng method to segment a cluster of t u mor -l i ke region.
Proc MICCAI-BRATS 2012
32
2 Hoo-Chang Shin
2 Methods
2.1 Segmentation of Edema by Sparse Coding
Student Version of MATLAB
Student Version of MATLAB
Flair T1 T1C T2
sigmoid function
0
1
0
0
sparse coding
sparse dictionary
Fig. 1. Sparse coding of a slice in a volume with its multi-modal MR images by a 4 ×4
size sparse dictionary.
A sparse dictionary for the behavior of the image intensities in the given
multiple image modalities (Flair, T1, T1C, and T2) is learned by a sparse au-
toencoder [10, 11]. The 4×4 sparse basis dictionary is shown in Fig. 1, where each
column represents a dictionary entry. The image intensities in a pixel position of
different image modalities are convolved with each of the dictionary entry, where
the values become a binary digit after b eing applied a logistic sigmoid function.
Different combinations of dictionary entries represent the characteristics of
different tissue types, which results in different binary digit numbers. 15(= 2
4
1)
different types of tissue characteristics can be captured by sparse coding with
a dictionary of 4 × 4 size. The hyper-parameters in sparse dictionary learning,
such as the size of the dictionary and the sparsity constraints, are chosen by
cross-validation.
The visualization of the 15 different tissue types identified by the sparse cod-
ing is shown in Fig. 2 with the ground truth of edema and tumor for the slice. It
is noticeable that edema is already quite well classified by the bright blue region
where it contains a region of a tumor as well. The segmentation performance for
edema by sparse c od in g in F1-score (2 · (precision · recall)/(precision + recall))
on 10 cross-vali d at ion set was 0.54, which outpe rf or med the other methods tried
for edema segmentation (logist i c regression: 0.246, neural network
1
: 0.14).
1
neural network classification on image intensities of each voxel in the
Proc MICCAI-BRATS 2012
33
Multi-Modal Bra in Tumor Segmentation 3
20 40 60 80 100 120 140 160 180 200
20
40
60
80
100
120
140
160
edema
tumor
Fig. 2. Tissue type clustering result of a n image slice by spa rse coding with 4 × 4 siz e
dictionary.
2.2 Segmentation of Tumor by Logistic Regression and K-means
Clustering
Whereas edema was well classified by the sparse cod i ng, tumor was not, probably
because tumor is a highly heterogeneous tissue type or just an abnormal region.
In the training dataset, tumor in some volumes were well observed by Flai r
images, whereas in some other volume it was better observed by T1-Contrast
images, but no obvious pattern could be found unlike the other tissue types or
edema. Therefore, it was tried to find a non-linear decision boundary to separ ate
these “abnormal” tumor from a more “normal ” tissue types with logistic regres-
sion. Logistic regression models the conditional probability of given feature x
belonging t o a class y ∈ {0, 1} as the logistic sigmoid function (1/(1 + e
x
)). Sec-
ond or d er polynomial feature with two combin ati on of the four image modal i t i es
were used
x = {x
1
x
2
, x
2
1
x
2
, x
1
x
2
2
, x
2
1
x
2
2
, x
1
x
3
, · · · , x
2
1
x
2
4
, x
3
x
4
, x
2
3
x
4
, x
3
x
2
4
, x
2
3
x
2
4
}
, wher e x
1
, x
2
, x
3
, x
4
represent the image intensities of each image modality. The
F1-score for tumor classification with this method was 0.246 on a cross-validation
dataset with 10 volumes, outperforming the other cl as si fic at ion methods for tu-
mor classification (sparse coding: 0.0031, neural network: 0.0001).
Another insight for the tumor segment at ion was, that a tumor is usually sur-
rounded by an edema. Consequently, if the region found by the logistic regression
is inside the already segmented edema region, it is regarded as well classified.
When the re gion segmented by logistic regression is outside of the edema region,
and then it is regarded as segmentation failure. In this case, a multi-modal k-
means clust er i ng was applied to capture a cluster of a region within edema but
has a different characteristic than edema.
Proc MICCAI-BRATS 2012
34
4 Hoo-Chang Shin
3 Results
The average segmentation performance evaluated are shown in Table 1. No hu-
man input is required durin g the segmentation process, and the average segmen-
tation time for a single pati ent volume was about 1.8 minutes.
Table 1. Performance evaluation results
Av.Dist.1 A v. Dis t .2 Dice.1 Dice.2 Haus.1 Hau s .2 Cohen’s Sens.1 Sens.2 S pec. 1 Spec.2
6.526 15.478 0.391 0.3 37 . 88 3 94.282 0.144 0.511 0.416 0.992 0.995
References
1. Lewicki, M.S. and Sejnowski, T.J.: Learning overcomplete representations. Neural
Computation, 1 2 , 337–365 (2000)
2. Kreutz-Delgado, K., Murray, J.F., Rao, B.D., Engan, K., L ee, T.W., Sejnowski, T.J.:
Dictionary learning algorithms for sparse representation. Neural Computation, 15,
349–396 (200 3 )
3. Ravishankar, S., Bresler, Y.: MR image reconstruction from highly undersampled
k-space data by dictionary learning, IEEE Transactions on Medical Imaging, 30,
1028–1041 (2011)
4. Elad, M., Aha ro n , M.: Image deno is in g via sparse and redundant rep res entations
over learned dictionaries, IEEE Transactions on Image Processin g , 1 5 , 373 6 – 3 7 4 5
(2006)
5. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robus t face recognition
via sparse representation. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 31, 210–227 (2009)
6. Zhang, S., Zhan, Y., Dewan, M., Huang, J., Metaxas, D., Zhou, X.: Deformable
segmentation via sparse shape representation. In Proceedings of MI CCA I 2011,
451–458 (201 1 )
7. Yang, J., Yu , K., Gong, Y., Huang, T.: Linear spatia l pyramid ma tching using sparse
coding for image classification, In Proceedin g s of CVPR 20 0 9 , 1794–1801, (2009)
8. Boureau, Y.L., Bach, F., LeCun, Y., Ponce, J.: Lea rn in g mi d -l evel feat u res for recog-
nition, In Proceedings of CVPR 2011, 2559–2566, (2010)
9. Shin, H.C., Orton, M., Collins, D.J., Doran, S., Leach, M.O.: Autoencoder in Time-
Series Analysis for Unsupervised Tissues Characterisation in a Large Unlabelled
Medical Image Dataset, In Proceedings of ICMLA 2011, 259–264 (2011 )
10. Marc’Aurelio, R., Boureau, L., and LeCun, Y.: Sparse Feature Lerning for Deep
Belief Networks, Advances in Neural Inform a t io n Processing S ys te ms , vol. 20 (2007)
11. Larochelle, H., Bengio, Y., Louradour, J., Lamblin, P.: Exp lo ri n g strategies for
training deep neural networks, The Journal of Machine Learning Research , 10, 1-40
(2009)
Proc MICCAI-BRATS 2012
35
Hierarchical Random Walker for Multimodal Brain Tumor
Segmentation
Yang Xiao, Jie Hu
School of biomedical engineering, Southern Medical University, Guangzhou, China
Abstract. In this text, a Random Walker (RW) based method is proposed for brain tumor
MR images segmentation with interaction. Not only the method is designed to achieve a
final segmentation result, but also it is a convenient tool for users to modify their results
iteratively. To overcome the shortcoming of typical RW algorithm, we extend RW to fea-
ture space for soft clustering, and then carry out pixel-wise segmentation in image
space. Proposed method is performed on multimodal brain MR images, including T2-
weighted, contrast enhanced T1-weighted, and FLAIR sequences.
Keywords: image segmentation, brain tumor, Random Walker, interactive, hierarchical
1 Introduction
Segmentation of brain tumor image is very important. Because the inter-patient varia-
tion of tumor shape, position, texture, and size is large, a robust fully-automatic method
is difficult to design. In addition, intervention is needed when automatic result is evalu-
ated and validated. Therefore, we focus on interactive algorithms.
Since the Graph Cuts (GC) algorithm is brought to attention
[1], a family of interactive
segmentation methods are proposed and of interest for recent decade, such as Random
Walker (RW)
[2] and Shortest Path (SP) [3], which provide another chance to modify the
segmentation results when automatic implementation is unavailable or incorrect, espe-
cially on medical images. In
[4], these popular interactive algorithms are unified into a
common optimization framework. However, RW is distinguished from others due to its
particular soft segmentation. In next section, RW will be discussed, and our improve-
ment of typical algorithm and its application for multimodal brain tumor MR will be illus-
trated.
2 Method
2.1 Typical Random Walker
The RW algorithm is formulated on a weighted graph and optimized by solving a Dirichlet
problem. The objective function is given by
Proc MICCAI-BRATS 2012
36
∈
−
= =
∑
(1)
where
is the target soft label, and
is similarity measure between pixel
and
,
and the constraints are set by interaction.
Furthermore, similarity measure
is often defined as
β
= − ∇ (2)
where
∇
denotes absolute distance in feature space between pixel
and
, and
β
is a parameter given by designer.
2.2 RW in feature space
Similarly, the discrete feature space is defined as an edge-weighted graph, and the objec-
tive function of clustering is given by
∈
−
= =
∑
(3)
where
represents clustering label, as well as
is similarity measure of distribution
density between sample
and
in feature space, which is defined by
α
= − ∇ (4)
where
∇
is defined as variation of distribution density in feature space, and
α
is a
parameter given by designer.
In typical RW, similarity measure is defined by its negative correlativity to absolute dis-
tance in feature space, which neglects prior label information provided by user. To mine
the prior information from the seeds as much as possible, we extend RW to feature
space as a soft semi-supervised clustering. When the clustering labels are obtained, we
will re-define the similarity measure as in (2) by relative distance between labels as fol-
low,
β
= − − (5)
2.3 Proposed Method
The whole workflow of proposed method is illustrated in Fig. 1. In the pre-processing
step, T2, CET1, and FLAIR sequences are filtered by Gaussian kernel. To initialize the hier-
archical RW, the user has to select two groups of seeds which locate tumor and edema
respectively, and the interface for interaction, where the user can input seeds in images
of any modal, is shown in Fig. 2.
Proc MICCAI-BRATS 2012
37
Fig. 1. Flow diagram of proposed method.
Fig. 2. Interface for selecting seeds.
After the initialization, the core algorithm of proposed method, which is hierarchical RW,
is performed, and then results are displayed in the interface for user’s iterative modifica-
tion.
RW is carried out in T2-FLAIR intensity space, as described in last sub-section, to identify
the abnormal (tumor and edema), and in T2-T1C intensity space to identify the tumor
respectively. In Fig. 3, the joint distribution density and results of clustering is shown.
Fig. 3. From left to right: density in T2-FLAIR space; soft clustering labels; and images labeled by cluster-
ing result.
The labels of clustering are utilized to calculate similarity measure of pixel-pair according
to (5), and typical RW in image space is employed to obtain segmentation labels, as
shown in Fig. 4. If the segmentation is not accepted, the user could iteratively revise the
result until satisfied.
Proc MICCAI-BRATS 2012
38
Fig. 4. Illustration of segmentation results in some slice: edema (left) and tumor (right).
3 Results
The proposed method was applied to 80 sets of brain tumor MR images, of which 12
groups failed due to high space complexity of RW. With enough interaction, a user could
gain any segmentation result by an interactive segmentation algorithm, so we evaluate
the results acquired from single interaction without any modification. The performance
of our method is illustrated in Table 1 and Table 2 where 7 segmentation performance
metrics are introduced.
Table 1. The mean of evaluation metrics.
Table 2. The standard deviation of evaluation metrics.
4 Discussion and Conclusion
We presented a hierarchical RW method which assists user to locate tumor and edema
interactively in dozens of seconds. The improved algorithm performed better than typi-
cal one, while the feature space is multi-dimensional.
The RW is optimized by solving a large sparse equation, which needs adequate memory
resource, and when the size of image is huge, a technique for linear equation is neces-
sary.
Proc MICCAI-BRATS 2012
39
References:
1. Boykov, Y.Y. and M.P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of
objects in N-D images. in Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International
Conference on. 2001.
2. Grady, L., Random walks for image segmentation. IEEE transactions on pattern analysis and ma-
chine intelligence, 2006. 28(11): p. 1768-1783.
3. Xue, B. and G. Sapiro. A Geodesic Framework for Fast Interactive Image and Video Segmentation
and Matting. in Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. 2007.
4. Couprie, C., et al., Power Watershed: A Unifying Graph-Based Optimization Framework. Pattern
Analysis and Machine Intelligence, IEEE Transactions on, 2011. 33(7): p. 1384-1399.
Proc MICCAI-BRATS 2012
40
Automatic Brain Tumor Segmentation based on
a Coupled Global-Local Intensity Bayesian
Model
Xavier Tomas-Fernandez
1
and Simon K. Warfield
1
Children’s Hospit a l Boston, Boston MA 021 1 5 , USA,
Xavier.Tomas-Fernadez@childrens.harvard.edu,
http://crl.med.harvard.edu/
Abstract. Localizing and quantifying the volume of brain tumor from
magnetic resonance images is a key task for the analysis of brain cancer.
Based in the well established global gaussian mix t u re model of brain tis-
sue segmentation, we present a novel algorithm for brain tissue segmen -
tation and brain tumor de te ct i on . We propose a tissue model which com-
bines the p a t ient global intens ity model with a population local intensity
model derived from an alig n ed reference of healthy subjects. We used the
Expectation-Maximization algorithm to estimat e the parameters which
maximize the tissue maximum a pos te rio ry probab i li ti es . Brain tumors
were modeled a s outliers wit h respect to our coupled global/local i nten-
sity model. Brain tumor segmenta t io n was va l id a t ed using the 30 glioma
patients scans from the training dataset from MICCAI BRATS 2012
challenge.
Keywords: Magnetic Resonance Imaging, Segmentation, B ra in Tumor,
Bayesian, Ou t li ers
1 Introduction
Quantifying the volume of a brain tumor is a key indicator of tumor progression.
The standard technique to obtain an accurate segmentation is through manual
labeling by experienced technicians. Howe ver, this manual process is not only
very time consuming, b u t prone to inherent inter- and intra-rater variability.
Therefore, there is a need for automat ed methods for brain tumor segmentation
that can analyse large amounts of data in a reproducible way and correlates well
with expert analyses.
Several approaches to br ai n tumor segmentat ion used intens ity based statisti-
cal classification algorithms. Kaus et al [3] used the adap t ive template-moderated
classification algorithm [9] to segment the MR image int o four healthy tissue
classes and tumor. Their technique proceeded as an iterative seq u en ce of spa-
tially varying class ifi cat i on and nonlinear registration. Prastawa et al. [7] detect
outliers based on r efi n ed intensity distributions for healthy brain tissue initially
derived from a registered probabilistic atlas.
Proc MICCAI-BRATS 2012
41
Intensity based classification rel i es on contrast between tissue types in f eat ur e
space and adequate signal compared to image noise. Statistical classification
identifies an optimal boundary, in feature space, between tissue types, and the
separability of two tissue c las se s is related to the ove r lap between classes i n
feature space. Brain tumor MRI intensity feature space overlap with those of
brain healthy tissues making difficult to identify a decision boundary, resulting
in increased tissue miscl as si fic at i on.
In this paper, we present a novel algorithm for brain tissue segmentati on
and brain tumor detection which extents the well e st ab l is he d global Bayesian
model of brain tissue segmentation that has been proposed previously in se veral
works (i.e. [10]). We propose a novel tissue model which combines a patient
global intensity model with a population local intensity model derived from a
reference of healthy subject s. We reformulate the challenge of det ec t i ng brain
tumors by modeling them as outliers with respect to our coupled global/loc al
intensity model derived from the target subject an d an aligned healthy reference
population.
2 Methods
In this section, we first introduc e a local intensity prior probabili ty estimated
from a reference healthy population. This local tissue intensity prior and a global
Baye s ia n model of b r ain tissue intensities will be the foundation of our coupled
local and global Bayesian tissue model which we introduce in Section 2.2. Fol-
lowing, the procedure to detect brain tumors, will be described i n Section 2.3.
2.1 Local Reference Population Bayesian Intensity Tissue Model
Consider a reference population P formed by R healthy subjects aligned to
the subject of i nterest. Each reference subject is composed by a multispectral
grayscale MRI V (i.e. T1w, T2w and FLAIR) and the corres ponding tissue seg-
mentation L (i.e. GM, WM and CSF), t hus P = {V, L} = {V
1
, ..., V
R
; L
1
, ..., L
R
}.
Each refer en ce grayscale MRI V
r
= {V
r1
, ..., V
rN
} is formed by a finite set of N
vox el s with V
ri
ǫR
m
. Also each reference tiss ue segmentation L
r
= {L
r1
, ..., L
rN
}
is formed by a finite set of N voxels with L
ri
= e
k
= {l
ri1
, ..., l
riK
} is a K-
dimensional vector with each compone nt l
rik
being 1 or 0 according whether v
ri
did or did not arise from the k
th
class.
At each voxel i, the reference population P intensity distribution will be
modeled as a Gaussian Mixture Model parametrized by ξ
i
= {π
Pi
; µ
Pi
, Σ
Pi
}.
Where π
Pi
, µ
Pi
and Σ
Pi
are respectively the population tissue mix t ur e vector,
the population mean intensity vector and the population intensi ty covariance
matrix at voxel i. Because {V, L} are observed variables, ξ
i
can be derived
using the following express i ons :
π
P ik
=
1
R
R
X
j=1
p(L
ij
= e
k
) (1)
Proc MICCAI-BRATS 2012
42
µ
P ik
=
P
R
j=1
V
ij
p(L
ij
= e
k
)
P
R
j=1
p(L
ij
= e
k
)
(2)
Σ
P ik
=
P
R
j=1
(V
ij
− µ
P ik
)
T
(V
ij
− µ
P ik
) p(L
ij
= e
k
)
P
R
j=1
p(L
ij
= e
k
)
(3)
where p(L
ij
= e
k
) i s the probability of voxel i of the j
th
reference subject bel ong
to tissue k given by L
j
.
2.2 Coupling Global and Local Models
Consider that the observed data Y an d h idd en l abels Z are described by a
parametric conditional probability density function defined by {ψ, ξ}, where
ψ parametrize a w it h i n subject global GMM and ξ parametrize a local GMM
derived from an aligned reference of R healthy subjects P.
Segmenting the observed data Y implies the estimation of parameters {ψ, ξ}.
log L
C
(ψ, ξ) = log (f (Y, Z|ψ, ξ))
= log
N
Y
i=1
K
X
k=1
f(Z
i
= e
k
|ψ
k
, ξ
ik
)f(Y
i
|Z
i
= e
k
, ψ
k
, ξ
ik
)
!
(4)
From Section 2.1, the local inten si ty model is just dependent on the aligned
reference population P = {V, L} which is observable. Thus, the parameter ξ
will be constant. We can rewrit e the complete log-likelihood as:
log L
C
(ψ, ξ) = log (f(Y, Z|ψ, ξ)) = log (f (Y, Z|ψ)f (Y, Z|ξ))
=
N
X
i=1
log
K
X
k=1
f(Z
i
= e
k
|ψ
k
)f(Y
i
|Z
i
= e
k
, ψ
k
)
!
+
N
X
i=1
log
K
X
k=1
f(Z
i
= e
k
|ξ
ik
)f(Y
i
|Z
i
= e
k
, ξ
ik
)
!
where e
k
is a K-dimensional vector with each component being 1 or 0 according
whether Y
i
did or did not arise from the k
th
class. Thus:
log L
C
(ψ, ξ) =
N
X
i=1
K
X
k=1
z
ij
(log (f (Z
i
= e
k
|ψ
k
) + f (Y
i
|Z
i
= e
k
, ψ
k
)))
+
N
X
i=1
K
X
k=1
z
ij
(log (f (Z
i
= e
k
and|ξ
ik
) + f (Y
i
|Z
i
= e
k
, ξ
ik
))) (5)
Because the underlyin g tissue segmentation Z is unknown, the EM algorithm
will be used to find the parameters that maximize the complete log-likelihood.
Proc MICCAI-BRATS 2012
43
E-Step: Because ξ is constant, the E-step of the coupl ed model requires
the computation of the conditional expectation of log (L
C
(ψ, ξ)) given Y, using
ψ
(m)
for ψ.
Q(ψ, ξ; ψ
(m)
, ξ
(m)
) ∼ Q(ψ; ψ
(m)
) = E
ψ
(m)
{log L
C
(ψ, ξ)|Y}
The complete log-likelihood is linear in the hidden labels z
ij
. The E-step
requires the calculation of the current conditional expectation of Z
i
given the
observed data Y:
E
ψ
(m)
(Z
i
= e
k
|Y, ξ
ij
) = f(Z
i
= e
j
|Y
i
, ξ
ij
, ψ
(m)
j
)
=
f(Y
i
|Z
i
= e
j
, ξ
ij
, ψ
(m)
j
)f(Z
i
= e
j
|ξ
ij
, ψ
(m)
j
)
P
K
k=1
f(Y
i
|Z
i
= e
k
, ξ
ik
, ψ
(m)
k
)f(Z
i
= e
k
|ξ
ik
, ψ
(m)
k
)
(6)
Note that:
f(Y
i
|Z
i
= e
j
, ξ
ij
, ψ
(m)
j
) ∼ f(Y
i
|Z
i
= e
j
, ψ
(m)
j
)f(Y
i
|Z
i
= e
j
, ξ
ij
)
f(Z
i
= e
j
|ξ
ij
, ψ
(m)
j
) ∼ f(Z
i
= e
j
|ψ
(m)
j
)f(Z
i
= e
j
|ξ
ij
)
M-Step: Because the local reference population model parameter ξ is con-
stant, the Maximization step will consist in the maximization of Q(ψ, ξ; ψ
(m)
, ξ
(m)
)
with respect to ψ, which results in the following update equations:
f(Z
i
= e
k
|ψ
(m+1)
k
) =
1
N
N
X
i=1
f(Z
i
= e
k
|Y
i
, ψ
(m)
, ξ
ik
) (7)
µ
(m+1)
k
=
P
N
i=1
Y
i
f(Z
i
= e
k
|Y, ψ
(m)
)
P
N
i=1
f(Z
i
= e
k
|Y, ψ
(m)
)
(8)
Σ
(m+1)
k
=
P
N
i=1
f(Z
i
= e
k
|Y, ψ
(m)
)(Y
i
− µ
(m)
k
)
T
(Y
i
− µ
(m)
k
)
P
N
i=1
f(Z
i
= e
k
|Y, ψ
(m)
)
(9)
2.3 Brain Tumor Detection
Intuitively from (5), the local intensity model downweights the likelihood of
those voxels with an abnormal intensity given the reference population. Because
brain tumors show an abnormal intensity level compared to a healthy su bject in
the same locat i on, we assume that tumors correspond with brain areas with low
likelihood, making feasible brain tumor detection as outliers toward our coupled
global/local intensity model. To extract the voxel outliers and to estimate the
parameters of t he different brain tissues in a robust way, we used the Trimmed
Likelihood Estimator (TLE).
The TLE was proposed by [6] as a modi fic ati on of the Maximum Likelihood
Estimator in order to be robust to the presence of outliers. Using the TLE, the
Proc MICCAI-BRATS 2012
44
complete log-likelihood (5) can be expressed as:
log L
C
(ψ, ξ) = log
h
Y
i=1
f(Y
ν(i)
, Z
ν(i)
|ψ, ξ
i
)
!
with ν = {ν(1), ..., ν(N )} being the corresponding permutation of i nd i ces sorted
by their probabili ty f (Y
ν(i)
, Z
ν(i)
|ψ, ξ
i
), and h is the trimmi n g parameter corre-
sponding to the percentage of val ue s included in the parameter estimation.
The trimming proportion h will define a set of outlier voxels C wh i ch among
others will encompass the brain tumors. Becaus e the set of out li e r voxels C
contained no only brain tumor but also false positives, a Graph-Cuts algorithm
[2] was used to segment C into tumors or false positives.
3 Results
We validated our brain t um or segmentation algorithm using the clinical train -
ing data provided in the MICCAI 2012 brain tumor segment ati on (BRATS)
challenge.
3.1 Reference Population
We collected data from 15 volunteers on a 3T clinical MR scanner from GE
Medical Systems (Waukesha, WI, USA) using an 8-channel receiver head coil
and three different pulse sequences: a T1-weighted MPRAGE (Magnetization
Prepared Rapid Acquisition Gradient Echo) sequence; a T2-weighted scan from
an FSE (Fast Spin Echo) sequence; an d a FLAIR scan, also run with an FSE
sequence. We acquire d the T1w seque nc e axially; the T2w and FLAIR sequences
were sagitally acquired. All sequences were acquired with a matrix size of 256x256
and a field of view of 28 cm . Sl ic e thi ckness was 1.3 mm for the T1w-MPRAGE
sequence; 1 mm f or the T2w-FSE sequence; and 2 mm for the FLAIR-F S E
sequence. The MPRAGE parameters were TR 10/TE 6 ms with a flip angle of
8. For t h e FSE, the paramenters were TR 3000/TE 140 ms with an echo train
length of 15.
After image acquisition, we aligned the T2w and FLAIR images to the T1w
scan. Next, we re-oriented these MR images to an axial orientation. Last, a
trained expert manually segmented the intra-cranial volume, CSF, GM and WM
tissues [5].
To achieve accurate alignment between healthy volunteer s and a gliobas-
toma patient we used a nonlinear re gi st r ati on algorithm proposed by [1], which
although not intrinsic to our method, was selected because it is robust in the
presence of WM lesions [8]. We note, however that other non-linear registration
approaches are applicable with our technique [4].
Since the intensity levels of the subject of interest and the reference popula-
tion need to be in a comparable range, we used a linear transformation to find a
match between the median int en si ty of each modality (of each reference subject)
and those found in the scans of the subject of interest.
Proc MICCAI-BRATS 2012
45
3.2 MICCAI BRATS 2012 Training Data
MICCAI brain tumor segmentation challenge training data consists of multi-
contrast MR scans of 30 glioma patients (both low-grade and high-grade , and
both with and without resection) along with exp er t annotat i ons for ”active tu-
mor” and ”edema”. For each patient, T1, T2, FLAIR, and post-Gadolinium
T1 MR images are available. All volumes were linearly co- re gis t e re d to the T1
contrast image, skull strip ped, and interpolated to 1mm isotr op ic resolution.
Each gliobastoma patient was segmented using the post -G adol i ni u m T1, T2
and FLAIR MR scans. Algorithm paramete r s were fixed to neighborhood ra-
dius R = 1 and trimming percentage h = 10%. The estimated global-local
outliers were classified as either tumor or edema using the graph-cu t s algorithm
described in Section 2.3. An exam pl e of the resulting segmentation can be found
in Figure 1.
Fig. 1. Exam p le of gliobastoma segmentation achieved by our combined global-local
model. Top, FLAIR coronal slices of High grade gliobastoma Case0003; Middle, tumor
manual segmentation provided by the challenge organizers; Bottom, automatic tumor
segmentation achieved by our combined global/local model.
The resulting gliobastoma segmentations were uploaded to the MICCAI
BRATS 2012 challenge
1
for evaluation. The online evaluation to ol computed
the dice score, jaccard score, specificity, sensitivity, average closest distance and
hausdorff distance for both edema and active tumor. Results achieved by our
combined global-local model can be found in Table 1.
1
https://vsd.bfh.ch/WebSite/BRATS2012
Proc MICCAI-BRATS 2012
46
Dice Jaccard Specificity Sensitivity Avg. Dist. Hausdorff Dist.
Tumor 0.43±0.29 0.32±0.25 1.00±0.01 0.47±0.32 6.10±12.24 26. 8 2 ±1 8 .2 6
Edema 0.5 5 ±0 . 2 8 0.43±0.27 1.00±0.00 0.58±0.31 6.83±12.83 32.49±22.58
Table 1. MICCAI BRATS 2012 training d a t a results R = 0 and h = 20%.
4 Discussion
Preliminary results achieved on the cli n ic al training data, show how our com-
bined model allows a highly specific tumor segmentation. However, in some sub-
jects, the heterogeneous inte ns ity distribut i on of active tumors across all MRI
modalities provided in the database, our algorithms fails to classify acc urat e l y
outlier voxels into active tumor and edema.
Overall, we proposed a novel fully automatic algorithm for segmenting MR
images of gliobastoma patients. This algorithm uses an approach that combines
global and local tissue intensity models derived from an aligned set of healthy
reference subjects. Specifically, tumors are segmented as outliers within the com-
bined lo cal /gl ob al intensity Gaussian Mixture Model (GMM). A graph-cuts al-
gorithm is used to classify the estimated coupled model outlier voxels into active
tumor and edema.
References
1. Commowick, O.: Creation et Utilisation d’Atlas Anatomiques Numeriques pour la
Radiotherapie (Design and Use of Anatomical Atlases for Radiotherapy). Ph.D.
thesis, Universite de Nice-Sophia Antipolis (2007)
2. Greig, D., Porteous, B.: Exact maximum a posteriori estimation for binary images.
of the Royal Statistical Society. Series 51, 271–279 (1989)
3. Kaus, M.R., Warfield, S.K., Nab avi, A., Black, P.M., Jolesz, F.a. , Kikini s, R.:
Automated segmentation of MR images of brain tum ors . R ad io l o gy 218(2), 586–
91 (Feb 2001)
4. Klein, A., Andersson, J., Ardekani, B.A., Ashburner, J., Avants, B ., Chiang , M.C.,
Christensen, G.E., Collins, D.L., Gee, J., Helli er, P., Song, J.H., Jenkinson, M.,
Lepage, C., Rueckert, D., Thompson, P., Vercauteren, T ., Woods , R.P., Mann,
J.J., Parsey, R.V.: Evaluation of 14 nonlinear deformation algorithms applied t o
human brain MRI registration. NeuroImage 46(3), 786–802 (Jul 2009)
5. Makris, N., Meyer, J.W., Bates, J.F., Yeterian, E.H., K en n ed y, D.N., Caviness,
V.S.: MRI-Based topographic parcellation of human cerebral white matter and
nuclei II. Rationale and applic a ti o n s with systematics of cerebral connectivity.
NeuroImage 9(1) , 18–45 (Jan 1999)
6. Neykov, N., Filz m o ser, P., Dimova, R., Neytchev, P.: Robust fitting of mixtures
using the trimmed likelihood estimator. Computational Statist i cs & Da ta Analysis
52(1), 299–3 0 8 (Sep 2007)
7. Prastawa, M., Bullitt, E., Ho, S.: A brain tumor segmentation framework based
on outlier detection. Medical Image Anal ysi s 8(3), 275–283 (2004)
Proc MICCAI-BRATS 2012
47
8. Suarez, R.O., Commowick, O., Prabhu, S.P., Warfield, S.K.: Automated delin-
eation of white matter fiber tracts with a mult ip l e region-of-interest approach.
NeuroImage 59(4 ) , 3690–3700 (Nov 2011)
9. Warfield, S.K., Kaus, M., Jolesz, F.A., Kikinis, R.: A d a p t ive, template moderated,
spatially varying statistical classification. Medical image ana lys is 4(1), 43–55 (Mar
2000)
10. Wells, W.M., Grimson, W.L., Kikinis, R., Jolesz, F.a.: Adaptive segmentation of
MRI data. IEEE transactions on medical imaging 15(4), 429–42 (Jan 1996)
Proc MICCAI-BRATS 2012
48
Segmenting Glioma in Multi-Modal Images
using a Generative Model for Brain Lesion
Segmentation
Bjoern H. Menze
1,2
, Koen Van Leemput
3
, Danial Lashkari
4
Marc-Andr´e Weber
5
, Nicholas Ayache
2
, and Polina Golland
4
1
Computer Vision Laboratory, ETH Zurich, Switzerland
2
Asclepios Research Project, INRIA Sophia-Antipolis, France
3
Radiology, Massachusetts General Hospita l, Harvard Medical School, USA
4
Computer Science and Artificial Intelligence Laboratory,
Massachusetts Institute of Technology, USA
5
Diagnostic Ra dio lo g y, Heidelberg University Hospital, Germany
1 Introduction
In this pap e r, we evaluate a fully automated method for channel-specific tumor
segmentation in multi-dimensional images proposed by us in [1]. The method
represents a tumor appearance model for multi-dimensional sequences that pro-
vides channel-specific segmentation of the tumor. Its generative model shares
information about the spatial location of the lesion among channels while mak-
ing full use of the highly specific multi-modal signal of the he al thy tissue classes
for segmenting nor mal tissues in the brain. In addition to tissue types, the model
includes a lat e nt variable for each voxel enco d i ng the probability of observing
tumor at that voxel, b ase d on the ideas from [2, 3]. This extends the general “EM
segmention” algori t hm for situations when specific spati al structures cannot be
described sufficiently through population priors. Different from [1], we now use
a simplified EM algorithm for estimating the tissue state that also allows us to
enforce additional constrai nts for segmenting lesions that are either hyper- or
hypo-intense with respect to other tissues visible in the same image .
2 Generative Tumor Model
We use a generative modeling approach, in which we first build an explicit sta-
tistical model of image formati on and subsequently use this model to derive a
fully automatic segmentation algorithm. We follow the description of the model
from [1].
Normal state We model the normal state of the healthy brain using a spatially
varying probabilistic prior π
k
for each of the K tissue classes. This population
prior (atlas) is estimated from pri or examples and is assumed to be known. At
each voxel i, the atlas defines a multinomial distribution for the tissue label k
i
:
p(k
i
= k) = π
ki
. (1)
Proc MICCAI-BRATS 2012
49
The normal state k
i
is shared among all C channels at voxel i. In our experiments
we assume K = 3, represent i ng gray mat t er (GM), white matter (WM) and
cerebrospinal fluid (CSF).
Tumor state We model the tumor state using a spatially varying “latent” prob-
abilistic atlas α, similar to [ 2] , that is specific to th e given image data set or
patient. At each voxel i, this atlas provides a scalar parameter α
i
that defines
the probability of observing tumor at that voxel. Parameter α
i
is unknown and
is estimated as part of the segmentation pr ocess. We define a latent tumor state
t
c
i
∈ {0, 1} that indicates the presence of tumor in channel c at voxel i and model
it as a Bernoulli random variable with parameter α
i
. We for m a binary tumor
state vector t
i
= [t
1
i
, . . . , t
C
i
]
T
indicating t he tumor presence for all c observations
at voxel i, with probability
p(t
i
; α
i
) =
Y
c
p(t
c
i
; α
i
) =
Y
c
α
t
c
i
i
· (1 − α
i
)
1−t
c
i
. (2)
Observation model Th e image observations y
c
i
are generated by Gaussian inten-
sity distributions for each of the K tissue classes and the C channels, with mean
µ
c
k
and variance v
c
k
, respect i vely. In tumor tissue (i.e. , if t
c
i
= 1) the normal obser-
vations are replaced by intensities from another set of channel-specific Gaussian
distributions with mean µ
c
T
and variance v
c
T
, representing the tumor class. Let-
ting θ denote the set of all mean and variance parameters, and y
i
= [y
1
i
, . . . , y
C
i
]
T
denote the vector of the inte ns ity observations at voxel i, we define th e data like-
lihood:
p(y
i
|t
i
, k
i
; θ) =
Y
c
p(y
c
i
|t
c
i
, k
i
; θ)
=
Y
c
N ( y
c
i
; µ
c
k
i
, v
c
k
i
)
1−t
c
i
· N (y
c
i
; µ
c
T
, v
c
T
)
t
c
i
, (3)
where N (· ; µ, v) is the Gaussian dist ri b ut i on with mean µ and variance v.
Joint model Finally, t he joint probability of the atlas, the latent tissue class and
the observed variables
p(y
i
, t
i
, k
i
; θ, α
i
) = p(y
i
|t
i
, k
i
; θ) · p(t
i
; α
i
) · p(k
i
) (4)
is the product of the compon ents defined in Eqs. (1-3).
3 Inference
Maximum Likelihood Parameter Estimation We seek Maximum Likelihood esti-
mates of the model parameters {θ, α}:
h
b
θ,
b
αi = arg max
hθ,αi
p(y
1
, . . . , y
N
; θ, α) = arg max
hθ,αi
N
Y
i=1
p(y
i
; θ, α),
Proc MICCAI-BRATS 2012
50
where N is the number of voxels i n the volume and
p(y
i
; θ, α) =
X
t
i
X
k
i
p(y
i
, t
i
, k
i
; θ, α) =
X
s
i
p(y
i
, s
i
; θ, α).
For summing over values of t
i
and k
i
in Eq. (4), we follow the same approach
as in [1], but – rather than summing over the two par amet e rs individually – we
now sum over tissue state vector s
i
that is obtained by expanding t
i
and k
i
into
one state vector. This state vector s indicates tumor s
c
i
= T in all ch an ne ls with
t
c
i
= 1, and normal tissue s
c
i
= k
i
for all other channels. As an example, with
t
i
= [0, 0, 1, 1] and k
i
= W M indicating tumor in channels 3 and 4 and a white
matter image intensity in all healt hy channels, we obtain a tissue state vector
s
i
= [W M, W M, T, T ]. Letting {
e
θ,
e
α} denote the current parameter estimates,
we can compute the posterior probability of any of the resulting K ∗ 2
C
tissue
state vectors s
i
that may characterize the multimod al image intensity pattern
observed in voxel i. Writing out the components of Eq. (4) we obtain for s
i
(using the corresponding t
i
(s
i
) and k
i
(s
i
) for simplicity of the not at i on) :
p(s
i
|y
i
;
e
θ,
e
α) ∝ π
ki
Y
c
N ( y
c
i
; ˜µ
c
T
, ˜v
c
T
)
t
c
i
α
t
c
i
i
· N (y
c
i
; ˜µ
c
k
, ˜v
c
k
)
1−t
c
i
(1 − α
i
)
1−t
c
i
(5)
As an addi t i onal constrai nt we onl y consider state vectors s
i
that are biologically
reasonable. We rule out , for example, state vectors that indicate at the same time
CSF and tumor, or that correspond to observing tumor-specific changes in the
T1gad channel (that is characteristic for the tumor core), whi le T2 and FLAIR
do not show tumor specific changes in the same location. Choosing appropriate
constraints reduces th e total number of states |S| to be summed over in Eq. 5
significantly. Similar to the double EM-type min ori z ati on from [1] – that updated
t
i
and k
i
iteratively – we arrive at closed-form update expressions that guarantee
increasingly b et t e r estimates of the mod el parameters. The updates are intuitive:
the latent tumor p ri or ˜α
i
is an average of the corr e sponding posterior estimated
and the intensity parameters ˜µ
c
k
and ˜v
c
k
are updated with the weighted statistics
of the data for the healthy tissues and for the tumor clas s. We iterate the estima-
tion of the parameters {
e
θ,
e
α} and the computation of the posterior probabilities
p(s
i
|k
i
, y
i
;
e
θ,
e
α) until convergence that is typically reached after 10-15 updates.
During the iterations we enforced th at tumor voxels are hyper- or hypo-intense
with respect to the current average µ
c
k
of the white matter tissue (hypo-intense
for T1, hyper-intense for T1gad, T2, FLAIR) by reducing the class weight for
observations that do not comply wit h this constraint, similar to [4].
Spatial regularization Little spatial context is used in the basic model, as we
assume the tumor state t
i
in e ach voxel to be independent fr om the state of
other voxels Eq. 3). It is only the atlas π
k
that encourages smooth classification
for the healthy tissue classes by imposing similar priors in neighbori ng voxels.
To encourage a simi lar smoothness of the tumor labels, we extend the latent
atlas α to include a Mar kov Random Field (MRF) prior, relaxi n g the MRF to
Proc MICCAI-BRATS 2012
51
a mean-field approximation with an efficient approximate algorithm. Different
from [1], we now use channel-specific regularization parameters β
c
that are all
in the range of .3 to 1.
Channel-specific tumor probabilities, and semantic interpretation Once we have
an estim at e of the model parameters {
b
θ,
b
α}, we can evaluate the probability that
tumor is visible in channel c of voxel i by summing over all the configurations t
i
for which s
c
i
= T or t
c
i
= 1, respectively:
p(t
c
i
= 1|y
i
;
b
θ,
b
α) = p(s
c
i
= T |y
i
;
b
θ,
b
α) =
X
s
i
δ(s
c
i
, T ) p( s
i
|y
i
;
b
θ,
b
α), (6)
where δ is the Kroneker delta that is 1 for s
c
i
= T and 0 otherwise.
We then assign channel c of voxel i to tumor if p(t
c
i
= 1|y
i
;
b
θ,
b
α) > 0.5.
For a semantic interpretation that is in line with the class definitions of t he
segmentation challenge, we label voxels that show tumor specific changes in the
T2 channel as edema, and voxels that show hyper-i ntense tumor specific changes
as tumor core. All other image voxels are considered to be normal. Moreover, we
remove any isolated regions that is smaller than .5 cm
3
in size.
4 Experiments
We evaluate our model on a the BRATS challenge data set of 25 patients with
glioma. The data set compr i se s T
1
, T
2
, FLAIR-, and post-Gadolinium T
1
MR
images, all images are s kul l str ip ped and co-re gi st er ed usi ng an affine registra-
tion. We segment the volume into the three healthy and an outlier class using a
freely avail abl e implementation of the EM segmentation with bias cor r ec t ion [5,
4]. Outliers are defined as being more than three standard deviations away from
the centr oi d of any of the three normal tissue classes. We apply our algori t hm to
the bias field corrected volumes and initialize intensity parameters with values
estimated in the in i t ial segmentation. We initialize the latent atlas α to 0.7 time
the local prior for the pres en ce of gray or white matter.
Channels-specific segmentations returned by our algorithm are transformed
to Edema and Core classes as detailed above. Exemplary segmentations are
shown in Figure 1 and quantitative results from a leave-one-out cross- vali dat ion
are shown in Table 1. Note that the definition of “core” labels differs between
ground truth (where it also includes the T1 hypo-intense center of the tumor) an d
the algori th m tested (where it is only the T1gad hyper-intense area of the tu mor )
leading to misleading evaluation scores for low-grade cases an d , to some degree,
for high-grade core labels. Pleas e note that another submis si on to the BRATS
challenge [6] deals with further processing the probability maps presented here.
Acknowledgements. This work was supported by the German Academy of Sciences
Leopoldina (Fellowship Pro g ram me LPDS 2009-10), the Aca d emy of Finland (133611),
INRIA Compu Tumor, NIH NIBIB NAM I C U54-EB005149, NIH NCRR NAC P41-
RR13218, NIH NINDS R01-NS051826 , NIH R01-N S 0 5 2 5 85 , NIH R01-E B0 0 6 7 5 8 , NIH
R01-EB009051, NIH P41-RR0140 75 and the NSF CAREER Award 0642971.
Proc MICCAI-BRATS 2012
52
ID Dice1 Sens 1 Spec1 Dice2 Sens2 Spec2
BRATS HG0027 0.633 0.649 0.995 0.728 0.619 0.999
BRATS HG0026 0.681 0.616 0.998 0.443 0.369 0.999
BRATS HG0025 0.643 0.704 0.996 0.154 0.087 1
BRATS HG0024 0.652 0.685 0.998 0.71 0.639 0.999
BRATS HG0022 0.689 0.683 0.998 0.463 0.311 1
BRATS HG0015 0.762 0.699 0.998 0.691 0.534 1
BRATS HG0014 0.217 0.453 0.989 0.457 0.303 1
BRATS HG0013 0.429 0.647 0.999 0.74 0.983 1
BRATS HG0012 0.373 0.58 0.997 0.068 0.043 1
BRATS HG0011 0.606 0.464 0.999 0.57 0.54 0.998
BRATS HG0010 0.381 0.792 0.996 0.724 0.77 1
BRATS HG0009 0.697 0.594 0.997 0.486 0.38 0.997
BRATS HG0008 0.652 0.56 0.996 0.697 0.556 1
BRATS HG0007 0.542 0.492 0.997 0.775 0.727 0.999
BRATS HG0006 0.649 0.621 0.997 0.65 0.505 1
mean 0.574 0.616 0.997 0.557 0.491 0.999
median 0.643 0.621 0.997 0.65 0.534 1
ID Dice1 Sens1 Spec1 Di ce 2 Sens2 Spec2
BRATS LG0015 0.373 0.523 0.998 0 0 1
BRATS LG0014 0.182 0.335 0.998 0 0 1
BRATS LG0013 0.185 0.324 0.995 0.17 0.099 1
BRATS LG0012 0.42 0.79 0.997 0.005 0.002 1
BRATS LG0011 0.344 0.777 0.996 0.001 0 1
BRATS LG0008 0.471 0.386 1 0.675 0.547 1
BRATS LG0006 0.625 0.809 0.998 0.591 0.507 1
BRATS LG0004 0.75 0.764 0.998 0.011 0.006 1
BRATS LG0002 0.584 0.622 0.991 0.109 0.059 1
BRATS LG0001 0.3 0.495 0.997 0.838 0.777 1
mean 0.423 0.582 0.997 0.24 0.2 1
median 0.396 0.572 0.998 0.06 0.0325 1
Table 1. Performance measures as returned by the online challenge tool
(challenge.kitware.com/midas/), indicating Dice score, sensitivity and speci-
ficity (top: high-grade cases; bot t om: low-grade cases). Class “1”, with results
shown in the left column, refers to the “edema” labels. Class “2”, wi t h results
shown in the ri ght column, refers t o the “tumor c ore ” labels (for both low and
high grade cases). Not e that this definition differs somewhat from the labels re-
turned by the algorithm that only indicate s T1gad hyper-i ntense regions as class
2, irrespectively of the grading (low/high) of the disease.
Proc MICCAI-BRATS 2012
53
Fig. 1. Representative results of the channel-wise tumor segmentation. Shown
are the MR images together with the mos t likel y tumor areas (outlined red).
The first four columns show T1, T1gad, T2 and FLAIR MRI, lesions are hyper-
intense with respect to the gray matter for T1gad, T2 and FLAIR, they are
hypo-intense in T1. The last two columns show the labels i nf er r ed from the
channel-specific tumor segmentation (column 5), and the ground truth (column
6). The examples show that expert annotation may be disputable in some case s
(e.g., rows 4, 5, 6).
Proc MICCAI-BRATS 2012
54
References
1. Menze, B.H., Van Leemput, K., Lashkari, D., Weber, M.A. , Ayache, N . , Golland,
P.: A generati ve model for brain tu mo r segmentation in multi-modal images. In:
Proc MICCAI. (2010) 151–159
2. Riklin-Raviv, T., Menze, B.H., Van Leemput, K., Stieltjes, B., Weber, M.A., Ayache,
N., Wells, W.M., G o ll an d , P.: Joint segmentation via patient-spe ci fi c latent ana t omy
model. In: Proc MICCAI-PMM I A (Workshop on Probabilistic Models for Medical
Image Analysis). (2009) 244–255
3. Riklin-Raviv, T., Van Leemput, K., Menze, B.H., Wells, 3rd, W.M., Golland, P.:
Segmentation of image ensembles via latent atlases. Med Image Anal 14 (2010)
654–665
4. Van Leemput, K., Maes , F., Vandermeulen, D., Colchester, A., Suetens, P.: Auto-
mated segmentation of multiple sclerosis lesions by model outlier detection. IEEE
T Med Imaging 20 (2001) 677–688
5. Van Leempu t , K., Maes, F., Vandermeulen, D., Suetens, P.: Automated model-
based bias field correction of MR images of the brain. IEEE T Med Imaging 18
(1999) 885–8 9 6
6. Menze, B.H. , Geremia, E., Ayache, N., Szekely, G.: Segmenting glio ma in multi-
modal images using a generative-discriminative model for brain lesion segmenta-
tion. In: Proc MICCAI-BRATS (Multimodal Brain Tumor Segmentation Chal-
lenge). (2012) 8 p
Proc MICCAI-BRATS 2012
55
Segmenting Glioma in Multi-Modal Images
using a Generative-Discriminative Model for
Brain Lesion Segmentation
Bjoern H. Menze
1,2
, Ezequiel Ger em ia
2
, Nicholas Ayache
2
, and Gabor Szekely
1
1
Computer Vision Laboratory, ETH Zurich, Switzerland
2
Asclepios Research Project, INRIA Sophia-Antipolis, France
1 Introduction
In this paper, we evaluate a generative-discriminative approach for multi-modal
tumor segmentation that builds – in its generative part – on a generative sta-
tistical model for tumor appearance in multi-dimensional images [1] by using
a “latent” tumor class [2, 3], and – in its discriminative part – on a machine
learning approach based on a random forest using long-range features that is
capable of learnin g the local appearance of brain lesions in multi-dimensional
images [4, 5]. The ap pr oach combines advantageous properties from both types
of learning algorithms: First, it extr act s tumor related image features in a robust
fashion that is invariant to relative intensity change s by relyi n g on a generative
model encoding prior knowledge on expected physiology and pathophysiologi-
cal changes. Second, it transforms image features extracted from the generative
model – representing tumor probabilities in the different image channels – to
an arbitrary image repr es entation desired by the human interpreter through an
efficient classification method that is capable of dealing with high-dimensional
input data and that returns the desired class probabilities. In the following, we
shortly describe th e generative model from [1], and input features and additional
regularization methods used similar to our ear l ie r discriminative model from [4].
2 Generative Tumor Model
We use a generative modeling approach, in which we first build an explicit sta-
tistical model of image formati on and subsequently use this model to derive a
fully automatic segmentation algorithm. We follow closely our description of the
method from [1]. The st ru ct u re of the generative probabilist i c model provides
strong priors on the expected image patterns. For segmenting magnetic reso-
nance (MR) images, it has the advantage that mod el parameters describing the
observed image intensities serve as nuisance parameters. This makes it robust
against tissue specific changes of t he image intensity, and the algorithm does
not depend on intensity cal i b rat i on methods – of t en required for learning ap-
proaches that use image intensities as input – that may be prone to errors in
the prese nc e of lesions that vary i n size and image intensity. Moreover, gener-
ative image appearance model can be combined with other parametr i c model s
Proc MICCAI-BRATS 2012
56
used, for example, for registration [6] or bias field correction [7], and even more
complex image modification t h at can be “regressed out” in the same way.
Tumor appearance model We mo d el the normal state of the healthy brain us-
ing a spatially varying p r obab i li s ti c prior π
k
, a standard population atlas, for
each of the K = 3 t is su e classes that are visible from the given images (gray
matter, white matter, and cerebrospinal fluid). The normal state k
i
is shared
among all C channels at voxel i. We model the tumor state using a spatially
varying “latent” probabilistic atlas α, similar to [2]. At each voxel i, this atlas
provides a scalar parameter α
i
that defines the probability of observing a tum or
transition at that voxel. Paramet e r α
i
is unknown and is estimated as part of
the segmentation process. We further assu me that image observations y
c
i
are
generated by Gaussian intensity distributions for each of the K tissue classes
and the C channels, with mean µ
c
k
and variance v
c
k
, respectively. If the image in
channel c shows a transition from normal tissue to tumor in voxel i (i.e., if tissue
state s
c
i
= T ), the normal observati on s are replaced by intensities from another
set of channel-specific Gaussian distributions with mean µ
c
T
and variance v
c
T
,
representing the tumor cl as s.
Biological constraints on the estimated parameters We seek Maximum Likeli-
hood estimates of the model parameters {θ , α} by estimating the tissue stat e
vector s
i
of every voxel i that indicates the type of tissue visible in the dif-
ferent image modalities. The vector has s
c
i
= T in all channels that show tu-
mor, and has value s
c
i
= k
i
in all channels that appear normal. With K = 3
tissues type s and C = 4 channels (for the given data), the cardinality of the
state vector is |s| = K ∗ 2
C
= 3 ∗ 2
4
= 48. However, plausibility constraints
on the expected tumor appearance in the different ch ann el s apply, for exampl e
ruling out tumor-induced intensity changes in T1gad unless the same location
also shows tumor-induced changes in both T2 and FLAIR, and only gray and
white matter being able to show tumor transitions, the number of possible tis-
sue states reduces to |s| = 7. We estimate the most likely state vector s
i
in a
standard expectation maximization procedure, s i mi lar to the “EM segmention”
algorithm, with iterative updates of the parameters {
e
θ,
e
α} and the poste ri or
probabilities p(s
i
|k
i
, y
i
;
e
θ,
e
α). Updates can be pe rf orm ed using intuitive closed-
form expressions: the latent tumor prior ˜α
i
is an average of the corresponding
posterior esti m ate d, and t h e intensity parameters ˜µ
c
and ˜v
c
are updated with
the weighted statistics of the data for the healthy tissues and for the tumor
class. During the iter at ion we enforced that tumor voxels are hyper- or hypo-
intense with respect to the current average gr ay value of the white matter tissue
(hypo-intense for T1, hyper-intens for T1gad, T2, FLAIR) si mi l ar to [8]. Also
we encour age smoothne s s of the tumor labels by extending the latent atlas α to
include a Markov Random Field (MRF) prior, relaxing the MRF to a mean-field
approximation with an efficient approxi m ate algorithm. Different from [1], we
now use channel-specific regularization parameters β that are all in the range of
.3 to 1. Typically convergence is reached after 10-15 updates.
Proc MICCAI-BRATS 2012
57
Channel-specific tu mor and ti s s ue probabilities Once we have an estimate of the
model par ame t e rs {
b
θ,
b
α}, we can evaluate the pr ob abi l i ty that tumor is visible in
channel c of voxel i by summing over all the configurations S
i
for which S
c
i
= T :
p(s
c
i
= T |y
i
;
b
θ,
b
α) =
X
t
i
δ(s
c
i
, T ) p( t
i
|y
i
;
b
θ,
b
α), (1)
where δ is the Kroneker delta that is 1 for s
c
i
= T an d 0 otherwise . The generat i ve
model retu rn s C tumor appearance map p(s
c
i
= T |y
i
;
b
θ,
b
α), one for each channel
of the input volume. It also returns the probability maps for the K healthy
tissues p(k
i
|y
i
;
b
θ,
b
α), with global estimates for each voxel i t hat are valid for all
C images.
3 Discriminative Lesion Model
The present generative mo d el returns probability maps for the healthy tissues,
and probability maps for the presences of characteristi c hypo- or hyper-intens
changes in each of the image volu mes . While this p rovides highly specific infor-
mation about different pathophysiological processes i nd uc ed by the tumor, the
analysis of the multimodal image s eq ue nc e may still require to highlight specific
structures of the lesion – such as edema, the location of the active or necrotic
core of the tumor, “hot spots” of modified angiogenesis or metabolism – that
cannot directly be associated with any of these basic parameter maps returned.
As a consequen ce, we p r opos e to use the probabilist i c output of the generative
model, together with few str u ct u ral features that are derived from the same
probabilistic maps, as input to a cl ass ifi er modeling the posterior of the desired
pixel classes. In this we follow the approach proposed by [4] that prove useful for
identifying white matter lesion in multiple input volumes. The building blocks of
this d is cr i mi nat i ve approach are the input features, the parametrization of the
random forest classifier used , and the final post-processing routines.
Image features As input feature describing the i mage in voxel i we use the
probabilities p(k
i
) for the K = 3 tissue classes (x
k
i
). We also use the tumor
probability p(s
c
i
= T ) for each channel C = 4 (x
c
i
), and the C = 4 image
intensities after calibrating them with a global factor that has been estimated
from gray and white matter tissue (x
im
i
). From these data we derive two types
of features: the “long range features” that calculate differences of local i mage
intensities for all three ty pe s of input features (x
k
i
, x
c
i
,x
im
i
), and a dist an ce
feature that calculat es the geodesic dist an ce of each voxel i to characteristic
tumor areas.
The first type of features calculate the difference between the image intensity,
or scalar of any oth er map, at voxel j that is locat ed at v and the im age inte ns ity
at another voxel k that is located at v + w (with v here being 3D spatial
coordinates). For every voxel j in our volume we calculate these differences
x
diff
j
= x
j
− x
k
for 20 different directi on s w around v with spatial offsets
Proc MICCAI-BRATS 2012
58
between 3mm to 3cm. To reduce noise the subtracted value at v +w is extracted
after smoothing the image intensities locally around voxel k (using a Gaussian
kernel with 3mm standard deviation).
The second type of features calculates the geodesic distance between the lo-
cation v of voxel j to specific image feature t hat are of particular interest in
the analysis. The path i s constrained to areas that are most likely gray matter,
white matter or tumor as predicted by the generat i ve model. More specifically,
we use the distance of x
δ t issu e
j
of voxel j to the boundary of the the br ai n tissue
(the interface of white and gray matter with CSF), and the distance x
δ ed ema
j
to the boundary of the T2 lesion representing the approximate location of the
edema. The second distance x
δ ed ema
j
is calculated ind ependently for voxels out-
side x
δ ed ema +
j
and inside x
δ ed ema −
j
the edema. In total we have 269 image fea-
tures x for each voxel when concatenating the vectors of x
k
, x
c
, x
im
, x
diff
, and
x
δ
.
Classifier and spatial regularization We use th e image features x defined above
to model the pr obab i li t i es p(L; x) of class labels L in the BRATS dat a set, and
the labels of the K normal tissue. For the normal classes (that are not available
from the manual annotation of the chal l en ge data set) we infer the maximum a
posterior estimates of the generative model and use them as label during training.
We choose random forests as our d is cr i mi n ative model as it uses labeled samples
as input and returns class probabilities. Random forests learn many decisi on
trees from bootstrapped samples of the training data, and at each split in the
tree they only evaluate a random subspaces to find the best split. The sp l i t
that separates samples of one class best against the others (with respect to Gini
impurity) is chosen. Trees are gr own until all nodes contain sample of a single
class only. In prediction new samples are pushed down all trees of the ensemble
and assigned, for each tr ee , to th e class of the terminal node they e nd up in .
Votes are averaged over all trees of the ensemble. The resulting normalized votes
approximate th e posterior for training samples that are independent of each other
[9]. To m in i mi ze correlation i n the traini ng data, and also to speed up training,
we draw no more 2000 samples from each of the ≈ 10
6
vox el s i n each of the 25
data set. We train an ensemble with 300 randomized decision trees, and ch oose
a subspace dimensionality of 10. We use the random forest implementation from
Breiman and Cutl er . To improve segmentation, we use a Markov Random Fie ld
(MRF) imposing a smoothness constraint on the class labels. We optimi z e the
function imposing costs when assigning different labels in a 6 neighbourhood on
the cross-validated predictions on the training data.
4 Experiments
We evaluate our model on the BRATS challenge data set of 25 patients with
glioma. The data set compr i se s T
1
, T
2
, FLAIR-, and post-Gadolinium T
1
MR
images, all images are skull stripped and co-registered. We segment the volume
Proc MICCAI-BRATS 2012
59
into the three healthy and an outlier class using a freely available implemen-
tation of the EM segmentation with bias correction [7, 8]. Outliers are defined
as being more than three standard deviations away from the centroid of any of
the three normal tissu e classes. We apply the generative model to the bias field
corrected volumes and initialize intensity parameters with values estimated in
the initial segmentation; we also use the the bias field an d intensity corrected
images as input for the discriminative model. More details about these data is
given i n another submissi on to the BRATS challenge that focuses on evaluating
the generative model [10].
Exemplary segme ntations that are r et u rn ed from the present approach are
shown in Figure 1 and quantitative results from a leave-one-out cross- vali dat ion
are shown in Table 1. Note that the definition of “core” labels differs between
ground truth (where it also includes the T1 hypo-intense center of the tumor )
and the algorithm tested (where it is only the T1gad hyper-intense area of the
tumor) which re su lt s in misleading evaluation scores for the “core” class in low-
grade cases.
Acknowledgements. This work was suppo rt ed by NCCR Co-Me of the Swiss Na-
tional Science Foundati o n , and INRIA CompuTumor.
References
1. Menze, B.H., Van L eem pu t, K. , La sh kari, D., Weber, M.A., Ayache, N., Golland,
P.: A generative model for brain tumor segmentation in multi-modal images. I n :
Proc MICCAI. (2010) 151–159
2. Riklin-Raviv, T., Menze, B.H., Van Leemput, K., Stielt jes, B., Weber, M.A., Ay-
ache, N., Wells, W.M., Golland, P.: Joint segmentation via pa t ie nt-specific latent
anatomy model. In: Proc MICCAI -P M M I A (Workshop on Probabilistic Models
for Medical Image Analysis). (2009) 244 – 2 5 5
3. Riklin-Raviv, T., Van Leemput, K., Menze, B.H., Wells, 3rd, W.M., Golland, P.:
Segmentation of image ens embles via latent at l a ses . Med I ma g e Anal 14 (2010)
654–665
4. Geremia, E., Menze, B.H., Clatz, O., Konukoglu, E., Criminisi, A., Ayache, N.:
Spatial decision forest s for MS lesion segmentation in multi-channel MR images.
In: Proc MICCAI. (2010)
5. Geremia, E., Clatz, O., Menze, B.H., Konukoglu, E., Criminisi, A., Ayache, N.:
Spatial decision fores ts for MS lesion segmentation i n multi-channel magnet ic res-
onance images. Neuro im a g e 57 (2011) 378–90
6. Pohl, K.M., Warfield, S.K., Kikinis, R., Grimson, W.E. L. , Wells, W.M.: Coupling
statistical segmentation and -pca- shape modelin g . In: Medical Image Comput-
ing and Computer-Assisted Intervention. Volume 3216/2004 of Lecture Notes in
Computer Scienc e., Rennes / St-Malo , France, Springer, Heidelberg (2004) 151 –
159
7. Van Leemput, K., Maes, F., Vand erme u len , D., Suetens, P.: Automated model-
based bias field correction of MR images of the brain. IEEE T Med Imaging 18
(1999) 885–8 9 6
Proc MICCAI-BRATS 2012
60
Fig. 1. Representative results of the tumor segmentation. Shown are the
maximum-a-posteriori (MAP) estimates as obt ai ne d from the random forest for
normal and tumor classe ) , the probabilities for core and edema (column 2,3),
the MAP estimates of the two tumor classes before and aft e r spatial smoothing
(column 4,5), and the ground truth (column 6). The examples show that expert
annotation may be di sp u tab l e in som e cases.
Proc MICCAI-BRATS 2012
61
ID Dice1 Sens 1 Spec1 Dice2 Sens2 Spec2
BRATS HG0027 0.735 0.823 0.995 0.822 0.898 0.997
BRATS HG0026 0.738 0.758 0.997 0.412 0.401 0.998
BRATS HG0025 0.641 0.934 0.992 0.06 0.031 1
BRATS HG0024 0.7 0.834 0.997 0.896 0.982 0.999
BRATS HG0022 0.779 0.806 0.998 0.821 0.729 1
BRATS HG0015 0.8 0.82 0.996 0.894 0.879 0.999
BRATS HG0014 0.327 0.476 0.994 0.761 0.696 0.998
BRATS HG0013 0.7 0.661 1 0.887 0.985 1
BRATS HG0012 0.629 0.704 0.999 0 0 1
BRATS HG0011 0.808 0.763 0.998 0.9 0.889 0.999
BRATS HG0010 0.664 0.788 0.999 0.836 0.879 1
BRATS HG0009 0.833 0.822 0.997 0.749 0.604 1
BRATS HG0008 0.784 0.679 0.999 0.917 0.979 0.998
BRATS HG0007 0.644 0.508 0.999 0.838 0.942 0.999
BRATS HG0006 0.7 0.795 0.994 0.793 0.731 0.999
mean 0.699 0.745 0.997 0.706 0.708 0.999
median 0.7 0.788 0.997 0.822 0.879 0.999
ID Dice1 Sens1 Spec1 Di ce 2 Sens2 Spec2
BRATS LG0015 0.402 0.751 0.997 0 0 1
BRATS LG0014 0.405 0.605 0.999 0 0 1
BRATS LG0013 0.29 0.492 0.996 0.164 0.089 1
BRATS LG0012 0.424 0.94 0.996 0 0 1
BRATS LG0011 0.3 0.908 0.994 0 0 1
BRATS LG0008 0.419 0.53 0.999 0.521 0.397 1
BRATS LG0006 0.767 0.992 0.998 0.788 0.74 1
BRATS LG0004 0.813 0.898 0.998 0 0 1
BRATS LG0002 0.652 0.767 0.989 0.017 0.009 1
BRATS LG0001 0.454 0.552 0.999 0.843 0.915 0.999
mean 0.493 0.744 0.996 0.233 0.215 1
median 0.422 0.759 0.998 0.009 0.005 1
Table 1. Performance measures as returned by the online challenge tool
(challenge.kitware.com/midas/)indicating Dice score , sensitivity and speci-
ficity (top: high-grade cases; bot t om: low-grade cases). Class “1”, with results
shown in the left column, refers to the “edema” labels. Class “2”, wi t h results
shown in the ri ght column, refers t o the “tumor c ore ” labels (for both low and
high grade cases). Not e that this definition differs somewhat from the labels re-
turned by the algorithm that only indicate s T1gad hyper-i ntense regions as class
2, irrespectively of the grading (low/high) of the disease.
Proc MICCAI-BRATS 2012
62
8. Van Leemput, K., Ma e s, F . , Vandermeulen, D., Colchester, A., Suetens, P.: Auto-
mated segmentation of multiple sclerosis lesions by model outlier detection. IEEE
T Med Imaging 20 (2001) 677–688
9. Breiman, L.: Random forests. Mach Learn J 45 (2001) 5–32
10. Menze, B.H., Va n Leem p u t, K. , La sh kari, D., Weber, M.A., Ayache, N., Golland,
P.: Segmenting glioma in multi-modal images using a generati ve model for brain
lesion segmentation. In: Proc MICCAI-BRATS (Multimodal Brain Tumor Seg-
mentation Challenge). (2012) 7p
Proc MICCAI-BRATS 2012
63
Multi-modal Brain Tumor Segmentation via
Latent Atlases
Tammy Riklin Ravi v
1,2,3
, Koen Van Leemput
4,5
, and Bjoern H. Menze
6,7
1
Psychiatry and NeuroImaging Laboratory, Harvard Medical School, B ri g h a m and
Women’s H o sp i t al , Boston, USA
2
Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, USA
3
The Broad Institute of MIT and Harvard, Cambridge, USA
4
Harvard Medi ca l School, Mass General Hosp it a l, Boston, USA
5
Technical University of Denmark, Lyngby, Denmark
6
Computer Vision Laboratory, ETH Zurich, Switzerland
7
Asclepios Research Project, INRIA Sophia-Antipolis, France
Abstract. In this work, a generative approach for patient-specific seg-
mentation of brain tumors across different MR modalities is presented. It
is based on the latent atlas approach presented in [7, 8]. The individual
segmentation of each scan supports the segmentation of the ensemble by
sharing common information. This common information, in the form of
a spatial probability map of the tumor locati on is inferred concurrently
with the evolution of the segmentations. The joint segmentation problem
is solved via a stat is ti c a ll y driven level-set framework. We illustrate the
method on an example applicatio n of multimodal and longitudinal brain
tumor segmentation, reporting promising segmentation results.
1 Introduction
Modeling patient-specific anatomy is essential in longitudinal studies and pathol-
ogy detection. We present a generative approach for joint segmentation of MR
scans of a specific subject, where the latent anatomy, in the f orm of spatial pa-
rameters is inferred concurrently with the segmentation. The work is based on
the latent atlas approach presented in [7, 8]. While the methodology can be
applied to a variety of applications, here we demonstrate our algorithm on a
problem of multimodal segmentation of brain tumors. Patient-specific datasets
acquired through different modalities at a particular time point are segmented
simultaneously, yet individually, based on the specific parameters of their in-
tensity distrib ut i on s. The spatial parameters that are shared among the scans
facilitate the segmentation of the group.
The method we propose is almost fully automatic. No prior knowledge or
external infor mat i on is required but a couple of mouse clicks at approximately
the center and the boundary of a single tumor slice used to generate a sphere
that initializes the segmentations. All model parameters, spatial and intensity,
are inferred from the patient scans alone. The output of the algorithm consist of
Proc MICCAI-BRATS 2012
64
individual segmentations for each modality. This i s in contrast to many discrim-
inative met hods, e.g., [9], that use multimodal datasets for multivariate feature
extraction, assuming spatial coherence of the tumor outlines in different image
modalities. Here we relax this assumption and search for systematic, structural
differences of the visibl e tumor volume acquired by different imaging protocols.
2 Problem definition and probabilistic mode l
This section summarizes the formulation of [6–8] for the j oint segmentation of N
aligned MR images . The input consists of N scans of a specific patient acq ui r ed
via different imaging protocols. Our objec t ive i s to extract a b rai n tumor that
may appear slightly differently across the images. Let I
n
: Ω → R
+
, be a gray
level image with V voxe l s, defined on Ω ⊂ R
3
and let Γ
n
: Ω → {0, 1} be the
unknown segmentation of the image I
n
, n = 1, . . . , N . We assume that each
segmentation Γ
n
is generated iid from a probability distribution p(Γ ; θ
Γ
) where
θ
Γ
is the set of the unknown spatial paramet er s . We also assume that Γ
n
gener-
ates the observed image I
n
, independently of all other i m age-s egme ntation pairs,
with probability p(I
n
|Γ
n
; θ
I,n
) where θ
I,n
are the parameters corresponding to
image I
n
. Sin ce the images are acquired by different imaging protocols we assign
a different set of intensity parameters to each of them. Our goal is to estimate
the segmentations Γ . This, however, cannot be accomplished in a straightfor-
ward manner since the model parameters are also unknown. We therefore jointl y
optimize Γ and Θ:
{
ˆ
Θ,
ˆ
Γ } = arg max
{Θ,Γ }
log p(I
1
. . . I
N
, Γ
1
. . . Γ
N
; Θ) (1)
= arg max
{Θ,Γ }
N
X
n=1
[log p(I
n
| Γ
n
; θ
I,n
) + log p(Γ
n
; θ
Γ
)] . (2)
We alternate between estimating the maximum a posteri or i (MAP) segmenta-
tions and updating the model parameters. For a give n setti n g of the model pa-
rameters
ˆ
Θ, Eq. (2) implies that the segmentations can be estimated by solvin g
N separate MAP probl em s:
ˆ
Γ
n
= arg max
Γ
n
[log p(I
n
| Γ
n
; θ
I,n
) + log p(Γ
n
; θ
Γ
)] . (3)
We then fix
ˆ
Γ and estimate the model parameters Θ = {θ
Γ
, θ
I,1
, . . . θ
I,N
} by
solving two ML problems:
ˆ
θ
I,n
= arg max
θ
I,n
log p(I
n
; Γ
n
, θ
I,n
), (4)
ˆ
θ
Γ
= arg max
θ
Γ
N
X
n=1
log p(Γ
n
; θ
Γ
). (5)
Proc MICCAI-BRATS 2012
65
3 Level-set framework
Now we draw the connection between the probabilistic model presented above
and a level-set framework for segmentation. Let φ
n
: Ω → R be the level-set
function associated with image I
n
. The zero level C
n
= {x ∈ Ω| φ
n
(x) = 0}
defines the interface that partitions the image space of I
n
into two disjoint regions
ω and Ω\ω. Similar to [4, 5] we define the level-set function φ
n
using the log-odds
formulation instead of the conventional signed distance function:
φ
n
(x) , ǫ l ogi t (p ) = ǫ log
p(x ∈ w)
1 − p(x ∈ ω)
= ǫ log
p(x ∈ ω)
p(x ∈ Ω \ ω)
, (6)
where p(x ∈ ω) can be viewed as the probability that the voxel in location x
belongs t o the foreground region. The constant ǫ determines the scal i ng of the
level-set function φ
n
with respect to the ratio of the p r obab i li t i es . The inverse
of the logit function for ǫ = 1 is the logistic function:
H
ǫ
(φ
n
) =
1
2
1 + tanh
φ
n
2ǫ
=
1
1 + e
−φ
n
/ǫ
. (7)
Note, that H
ǫ
(φ
n
) is similar, though not i d entical, to the r egu l ari z ed Heaviside
function introduced by Chan and Vese [1]. We use this form of Heaviside function
and its derivative with respect to φ in the proposed level - set formulation. To
simplify the notation, we omit the subscript ǫ in the rest of the paper.
Cost functional for segmentation The joint estimation problem of the hidden
variables Γ and the unknown model paramete r s {θ
Γ
, θ
n
I
} can be solved as an
energy minimization problem. As in [6–8], we establish the correspondence be-
twe e n the log pr ob abi l i ty and the leve l-set ene rgy terms. We also look for the
fuzzy labeling functions H(φ
n
) rather than the hard segmentations Γ
n
.
Let us consider first the prior probability p(Γ
n
; θ
Γ
) in Eq. (2) and its corre-
sponding energy terms. Specifically, we construct an MRF prior for segmenta-
tions:
log p(Γ
n
; θ
Γ
) =
V
X
v =1
[Γ
v
n
log(θ
v
Γ
) + (1 − Γ
v
n
) log(1 − θ
v
Γ
)] (8)
−
V
X
v =1
f(Γ
v
n
, Γ
N (v)
n
) − log Z(θ
Γ
),
where Z(θ
Γ
) is the partition function and N (v) is the set of the closest neighbors
of voxel v. We define the spatial energy term E
S
based on the s in gl et on term in
Eq. (8). Using the level-set formulation we obtain:
E
S
(φ
n
, Θ) = −
Z
Ω
[log θ
Γ
(x)H(φ
n
(x)) + log(1 − θ
Γ
(x)) (1 − H(φ
n
(x)))] dx.
(9)
Proc MICCAI-BRATS 2012
66
The dynamically evolving lat ent atlas θ
Γ
is obtained by optimizing the sum of
the energy terms that depend on θ
Γ
:
ˆ
θ
Γ
(x) =
1
N
N
X
n=1
˜
H(φ
n
(x)). (10)
The standard smoothness te rm used in level-set framework:
E
LEN
(φ
n
) =
Z
Ω
|∇H(φ
n
(x))|dx, (11)
can be obtained as an approximation of the pairwise term in Eq. (8) .
The energy term E
I
(φ
n
, θ
n
I
) corresponds to the image likelihood term i n
Eq. (3):
E
I
(φ
n
, Θ) = −
Z
Ω
log p
in
(I
n
; θ
in
I,n
)H(φ
n
(x)) (12)
+ log p
out
(I
n
; θ
out
I,n
) (1 − H(φ
n
(x)))
dx.
We assume that the intensities of the structure of interest are drawn from a
normal distribution such t hat the pair of scalars θ
in
I,n
= {µ
in
n
, σ
in
n
} are the mean
and standard deviation of the foreground intensities. We use a local-intensity
model for t h e background intensity distributions in the spirit of [3], where
θ
out
I,n
(x) = {µ
out
n
(x), σ
out
n
(x)} are the local mean and standard deviation of a
small neighbourhood of x that exclusively belongs to the background.
We construct the cost functional for φ
1
. . . φ
N
and the mode parameters by
combing Eq. (12), (11) and (9):
E(φ
1
. . . φ
N
, Θ) = γE
LEN
+ βE
I
+ αE
S
(13)
where α, β and γ are positive scalars.
Gradient descent and parameter estimation. We optimize Eq. (13) by a
set of alternat i ng steps. For fixed model parameters the update of each level-set
function φ
n
in each iteration is determined by the following gradient descent
equation:
∂φ
n
∂t
= δ(φ
n
)
γ div (
∇φ
n
|∇φ
n
|
) + β [log p
in
(I
n
(x); θ
I,n
) − log p
out
(I
n
(x); θ
I,n
)]
+ α [log θ
Γ
− log(1 − θ
Γ
)]} , (14)
where δ(φ
n
) is the derivative of H(φ
n
) with respect to φ
n
. For fixed segmenta-
tions φ
n
, the model parameters are re covered by differentiating the cost func-
tional in Eq. (13) with res pect to each parameter.
Proc MICCAI-BRATS 2012
67
4 Experiments
We evaluate our model on the BRATS challenge data set of 25 patients with
glioma. The data set compr i se s T
1
, T
2
, FLAIR-, and post-Gadolinium T
1
MR
images, all image s are skull stripped and co-registered. The tumor is i n i t ial i ze d
through a sphere of 1-3 cm diameter, t h at is placed in the center of the tumor.
Exemplary segme ntations that are r et u rn ed from the present approach are
shown in Figure 1 and quantitative results from a leave-one-out cross- vali dat ion
are shown in Table 3. Note that the definition of “core” labels differs between
ground truth (where it also includes the T1 hypo-intense center of the tumor )
and the algorithm tested (where it is only the T1gad hyper-intense area of the
tumor) which re su lt s in misleading evaluation scores for the “core” class in low-
grade cases.
5 Discussion and future directions
We presented a statistically driven level-set approach for joint segmentation of
subject-specific MR scans. The latent patient anatomy, which is represented
by a set of spatial parameters is infer re d from the data simultaneously with
the segmentation through an alternating minimization procedure. Segmentation
of each of the ch an ne ls or modalities is th er ef ore supported by the common
information share d by the group. Promising segmentation results on scans of 25
patients with Glioma were demonstrated.
Acknowledgements. This work was suppo rt ed by NCCR Co-Me of the Swiss Na-
tional Science Foundati o n , and INRIA CompuTumor.
References
1. T.F. Chan and L.A. Vese. Acti ve c o ntours without edges. IEEE TIP, 10(2):266–277,
2001.
2. L. Dice. Measure of the amount of ecological associa ti o n between species. Ecology,
26(3):297–302, 1945.
3. S. Lankton and A. Tannenbaum. Local iz in g Region-Ba sed Active Contours. IEEE
Transactions on Image Processing,17 ( 1 1 ): 2 0 2 9 –2 0 3 9 , 2008.
4. K.M. Pohl and R. Kikinis and W.M. Wells. Active Mean Fields: Solving the Mean
Field Approximation in the Level Set Framework. IPMI, 4584:26–37, 2007.
5. Pohl et al. Using the logarithm of odds to defin e a vector space on probabilistic
atlases. Medical Image Analysis, 11(6):465–477, 2007.
6. T. Riklin Raviv, K. Van Leemput, W.M. Wells, and P. Golland. Joint segmentation
of image ensembles via la te nt atlases. In Medical Image Computing and Computer-
Assisted Intervention, pages 272–280, 2009.
7. T. Riklin Raviv, B.H . Menze, K. Van Leemput, B. Stieltjes, N. Weber, M.A. Ayache,
W.M. Wells, and P. Golland. Joint segmentation via patient-specific latent anatomy
model. In Proc. of MICCAI Workshop on Probabilistic Models for Medical Image
Analysis, 2009.
Proc MICCAI-BRATS 2012
68
Fig. 1. Representative results of the tumor segmentation. Shown are the s eg-
mentations for the four different modaliti e s (columns 1-4), the labels inferred
from the channel-wise segment at i on (column 5), and the ground truth (column
6). The examples show that expert annotation may be disputable in some cases.
Proc MICCAI-BRATS 2012
69
ID Dice1 Sens 1 Spec1 Dice2 Sens2 Spec2
BRATS HG0027 0.65 0.752 0.993 0.813 0.812 0.998
BRATS HG0026 0.678 0.602 0.998 0.413 0.293 0.999
BRATS HG0025 0.59 0.933 0.991 0.083 0.043 1
BRATS HG0024 0.659 0.873 0.996 0.825 0.779 1
BRATS HG0022 0.699 0.82 0.997 0.608 0.498 0.999
BRATS HG0015 0.756 0.908 0.991 0.831 0.731 1
BRATS HG0014 0.27 0.665 0.987 0.59 0.45 0.999
BRATS HG0013 0.684 0.713 1 0.894 0.996 1
BRATS HG0012 0.637 0.709 0.999 0.098 0.077 1
BRATS HG0011 0.798 0.742 0.998 0.882 0.971 0.998
BRATS HG0010 0.097 0.145 0.997 0.276 0.945 0.996
BRATS HG0009 0.795 0.8 0.995 0.548 0.377 1
BRATS HG0008 0.734 0.771 0.992 0.841 0.885 0.998
BRATS HG0007 0.407 0.361 0.996 0.278 0.298 0.996
BRATS HG0006 0.648 0.843 0.991 0.817 0.716 1
mean 0.607 0.709 0.995 0.586 0.591 0.999
median 0.659 0.752 0.996 0.608 0.716 0.999
ID Dice1 Sens1 Spec1 Di ce 2 Sens2 Spec2
BRATS LG0015 0.37 0.712 0.997 0.116 0.066 1
BRATS LG0014 0 0 1 0 0 1
BRATS LG0013 0.326 0.631 0.995 0.452 0.293 1
BRATS LG0012 0.563 0.721 0.999 0.822 0.762 1
BRATS LG0011 0.262 0.958 0.993 0 0 1
BRATS LG0008 0 0 1 0 0 1
BRATS LG0006 0.556 0.985 0.996 0.73 0.832 1
BRATS LG0004 0.513 0.492 0.997 0.022 0.072 0.997
BRATS LG0002 0.636 0.734 0.989 0.242 0.178 0.997
BRATS LG0001 0.345 0.648 0.997 0.843 0.774 1
mean 0.357 0.588 0.996 0.323 0.298 0.999
median 0.358 0.68 0.997 0.179 0.125 1
Table 1. Real data. Performance measures as returned by the online chal-
lenge t ool (challenge.kitware.com/midas/)indicating Dice score, sensitivity
and specificity (top: high-grade cases; bot tom : low-grade cases). Class “1”, with
results shown in the left column, refers to t h e “edema” labels. Class “2”, with
results shown in the right column, refers to the “tumor core” labe l s (for both
low and high grade cases). Note that this definition differs somewhat from the
labels returned by the algorithm that only indicates T1gad hyper -i ntense regions
as class 2, irrespectively of the grading (low/high) of the dise ase .
Proc MICCAI-BRATS 2012
70
ID Dice1 Sens 1 Spec1 Dice2 Sens2 Spec2
SimBRATS HG0025 0.023 0.105 0.996 0.631 0.593 0.998
SimBRATS HG0024 0.217 0.993 0.999 0.968 0.948 1
SimBRATS HG0023 0.007 0.078 0.997 0.454 0.419 0.997
SimBRATS HG0022 0.689 0.682 0.996 0.002 0.001 0.999
SimBRATS HG0021 0.312 0.214 0.998 0.02 0.011 1
SimBRATS HG0020 0.138 0.127 0.996 0.315 0.249 0.998
SimBRATS HG0019 0.45 0.349 0.998 0 0 1
SimBRATS HG0018 0.01 0.047 0.996 0.579 0.552 0.997
SimBRATS HG0017 0.147 0.179 0.998 0.499 0.348 1
SimBRATS HG0016 0.033 0.091 0.995 0.681 0.667 0.997
SimBRATS HG0015 0.36 0.289 0.998 0.234 0.186 0.998
SimBRATS HG0014 0.362 0.3 0.998 0.451 0.406 0.998
SimBRATS HG0013 0.623 0.564 0.996 0.004 0.002 0.999
SimBRATS HG0012 0.44 0.36 0.999 0.035 0.022 1
SimBRATS HG0011 0.453 0.518 0.997 0.366 0.235 1
SimBRATS HG0010 0.528 0.867 0.999 0.974 0.978 1
SimBRATS HG0009 0.762 0.788 1 0.958 0.977 1
SimBRATS HG0008 0.381 0.352 0.996 0.454 0.386 0.999
SimBRATS HG0007 0.635 0.689 0.995 0.559 0.75 0.997
SimBRATS HG0006 0.011 0.037 0.998 0.373 0.274 0.999
SimBRATS HG0005 0.63 0.615 0.996 0.019 0.015 0.999
SimBRATS HG0004 0.33 0.311 0.996 0.485 0.475 0.998
SimBRATS HG0003 0.63 0.593 0.998 0.317 0.314 0.999
SimBRATS HG0002 0.405 0.819 0.999 0.924 0.875 1
SimBRATS HG0001 0.592 0.856 0.999 0.971 0.982 1
mean 0.367 0.433 0.997 0.451 0.427 0.999
median 0.381 0.352 0.998 0.454 0.386 0.999
Table 2. Simulated data (high grade). Performance measures as returned
by the online challenge tool (challenge.kitware.com/midas/)indicating Dice
score, sensitiv i ty and specificity (top: high- grad e cases; bottom: low-grade cases).
Class “1”, with results shown in the left column, refers to the “ede ma” labels.
Class “2”, with results shown in the right column, refers to the “tumor core”
labels (for both low and high grade cases ). Note that this definition differs some-
what from the labels returned by the algorithm that only indicates T1gad hype r-
intense regions as class 2, irrespectively of the grading (low/high) of the disease.
Proc MICCAI-BRATS 2012
71
ID Dice1 Sens 1 Spec1 Dice2 Sens2 Spec2
SimBRATS LG0025 0.042 0.528 0.993 0.05 0.026 1
SimBRATS LG0024 0.404 0.997 0.993 0.137 0.074 1
SimBRATS LG0023 0.662 0.74 0.997 0.008 0.004 1
SimBRATS LG0022 0.404 0.551 0.997 0.007 0.004 1
SimBRATS LG0021 0 0 1 0 0 1
SimBRATS LG0020 0.367 0.702 0.997 0.023 0.012 1
SimBRATS LG0019 0.378 0.367 0.998 0.015 0.008 1
SimBRATS LG0018 0 0 1 0 0 1
SimBRATS LG0017 0.632 0.678 0.998 0 0 1
SimBRATS LG0016 0.699 0.858 0.993 0.08 0.045 1
SimBRATS LG0015 0.46 0.578 0.997 0.01 0.005 1
SimBRATS LG0014 0.02 0.416 0.996 0.025 0.013 1
SimBRATS LG0013 0.402 0.447 0.997 0.014 0.007 1
SimBRATS LG0012 0 0 1 0 0 1
SimBRATS LG0011 0 0 1 0 0 1
SimBRATS LG0010 0.078 0.694 0.991 0.21 0.117 1
SimBRATS LG0009 0.394 0.507 0.997 0.035 0.018 1
SimBRATS LG0008 0.051 0.998 0.994 0.13 0.07 1
SimBRATS LG0007 0 0 1 0 0 1
SimBRATS LG0006 0.395 0.857 0.995 0.054 0.028 1
SimBRATS LG0005 0.483 0.994 0.993 0.089 0.047 1
SimBRATS LG0004 0.317 0.316 0.998 0.002 0.001 1
SimBRATS LG0003 0.359 0.546 0.997 0.007 0.004 1
SimBRATS LG0002 0 0 1 0 0 1
SimBRATS LG0001 0.489 0.39 0.998 0.001 0.001 1
mean 0.281 0.487 0.997 0.0359 0.0194 1
median 0.367 0.528 0.997 0.01 0.005 1
Table 3. Simulated data (low grade). Performan ce measures as returned
by the online challenge tool (challenge.kitware.com/midas/)indicating Dice
score, sensitiv i ty and specificity (top: high- grad e cases; bottom: low-grade cases).
Class “1”, with results shown in the left column, refers to the “ede ma” labels.
Class “2”, with results shown in the right column, refers to the “tumor core”
labels (for both low and high grade cases ). Note that this definition differs some-
what from the labels returned by the algorithm that only indicates T1gad hype r-
intense regions as class 2, irrespectively of the grading (low/high) of the disease.
Proc MICCAI-BRATS 2012
72
8. T. Riklin Raviv, K. Van Leemput, B.H. Menze, W.M. Wells, and P. Golland. Seg-
mentation of i ma g e ensembles via latent atlases. Medical Image Analysis, 14(5):654–
665, 2010.
9. Van Leemput et al. Automated segmentation of multiple sclerosis lesions by model
outlier detection . IEEE TMI, 20:677–688, 20 0 1 .
Proc MICCAI-BRATS 2012
73
