ArticlePDF Available

Three- to five-dimensional biomedical multisensor imaging for the assessment of neurological (dys)function

Authors:
  • Imaging & Technology

Abstract and Figures

This report describes techniques and protocols implemented at the Geneva Canton University Hospitals (HUG) for the combination of various biomedical imaging modalities and sensors including electromagnetic tomography, to study, assess, and localize neurological (dys) function. The interest for this combination stems from the broad variety of information brought out by (functional) magnetic resonance imaging, magnetic resonance spectroscopy, computed tomography, single-photon emission tomography, positron emission tomography, and electromagnetic tomography. Combining these data allows morphology, metabolism, and function to be studied simultaneously, the complementary nature of the information from these modalities becoming evident when studying pathologies reflected by metabolic or electrophysiologic dysfunctions. Compared with other current multimodality approaches, the one at the HUG is totally compatible with both clinical and research protocols, and efficiently addresses the multidimensional registration and visualization issues. It also smoothly integrates electrophysiology and related data as fully featured modalities.
Content may be subject to copyright.
Three- to Five-Dimensional Biomedical Multisensor Imaging
for the Assessment of Neurological (Dys)function
Luc M. Bidaut, Roberto PascuaI-Marqui, Jacqueline Delavelle, Alain Naimi, Margitta Seeck,
Christophe Michel, Daniel Slosman, Osman Ratib, Daniel Ruefenacht, Theodor Landis,
Nicolas de Tribolet, Jean-Raoul Scherrer, and Fran~ois Terrier
This report describes techniques and protocols imple*
mented at the Geneva Canton University Hospitals
(HUG) for
the combination of various biomedical imag-
ing modalities and sensors including electromagnetic
tomography, to study, assess, and Iocalize neurologi-
cal (dys)function. The interest for this combination
stems from the broad variety of information brought
out by (functional) magnetic resonance imaging, mag-
netic resonance spectroscopy, computed tomography,
single-photon emission tomography, positron emis-
sion tomography, and electromagnetic tomography.
Combining these data allows morphology,
metabo-
lism,
and function
to be
studied simultaneously, the
complementary nature of the information from these
modalities becoming evident when studying
patholo-
gies
reflected by metabolic or electrophysiologic dys-
functions. Compared with other current multimodality
approaches, the one at the HUG is totally compatible
with both
clinical and research protocols, and effi-
ciently addresses the multidimensional registration
and visualization issues. It also smoothly integrates
electrophysiology and related data as fully featured
modalities.
Copyright 9 1996 by W.B.
Saunders Company
KEY WORDS: medical imaging (MI), multimodality
(MM) MI, multisensor (MS) MI, multidimensional MI,
three-dimensional MI, brain function, epilepsy, comput-
ers, radiology.
T
HE WORLD of biomedical imaging can be
divided in two major tracks: one morpho-
logical, where the images show the underlying
anatomic structures; the other metabolic or
functional, where the images produced provide
information about the metabolism or the func-
tion of the underlying tissues. The first track
includes x-ray computed tomography (CT) and
magnetic resonance imaging (MRI). The sec-
ond track includes single-photon emission com-
puted tomography (SPECT), positron-emission
tomography (PET), magnetic resonance spec-
troscopy (MRS), and functional MRI (fMRI).
The two tracks provide information that has
clearly been recognized as complementary and
may consequently often benefit from direct
comparison in various protocols or pathologies.
For example, multimodality (MM) approaches
are now well accepted in oncology, either for
the brain or other tissues. 1-5 Beyond tumors,
studies are also under way to investigate brain
function through MM approaches, 6,7 and other
pathologies or organs may also benefit from
such approaches. TM
As the modalities making up the two tracks
include both direct imaging and pseudoimaging
modalities (eg, parametric imaging), a more
generic name had to be coined and was chosen
as "sensor" by analogy with space- and air-
based remote sensing imaging concepts. In the
rest of this report, modality and sensor are
considered to have rather similar meanings, the
latter being considered to also include nondi-
rect imaging "modalities."
Because of the spreading need for multimo-
dality/multisensor (MS) approaches, and since
most modalities/sensors are still not produced
on the same machine or in the same spatial
referential, many groups have been involved in
three-dimensional (3D) data sets registration
issues so as to be able to work with different
modalities in a common space. 12 Approaches
used for registering data sets can either be
prospective or retrospective. Prospective ap-
proaches often imply artificial markers or fidu-
cials placed on the patient before the acquisi-
tions. In these approaches, the markers, not the
actual anatomy, are being aligned by the regis-
tration process. Retrospective techniques can
make use of routinely acquired data and, ide-
ally, do not imply a priori knowledge about any
subsequent MM alignment. Data sets are ac-
quired normally, without additional locator de-
vices. Most often, similar anatomic structures
are extracted from the initial data sets, and their
3D distance is minimized by iterative manipula-
tion of the 3D interset geometry transformation
From the Departments of Medical Inforrnatics, Neurology,
Radiology, Nuclear Medicine, and Neurosurgery, Geneva Can-
ton University Hospital, Geneva, Switzerland.
Address reprint requests to Luc Bidaut, PhD, UIN-DIM-
Radiology, Geneva Canton University Hospital, 24 Rue Micheli-
du-Crest, CH-1211 Genera 14, Switzerland.
Copyright 9 1996 by W.B. Saunders Company
0897-1889/96/0904-000553. O0 / 0
JournalofDigita/Imaging,
Mol 9, No 4 (November), 1996: pp 185-198 185
186 BIDAUT ET AL
(eg, through 3D translations and rotations).
The manipulation of the data sets to reach the
optimal alignment transformation can range
from direct manual interaction, to point-to-
point correspondence, 13,14 principal axes regis-
tration, 15 3D surfaces alignment, ~6-18 and also
3D volume correlation, a9 None of these ap-
proaches is general enough to take the best care
of every MM registration problem, and this is
the reason why there is continuous development
in this field. For example, different graphical
primitives can be used simultaneously in the
computation of the 3D distance between two
data sets to improve the result of the registra-
tion. 2o Other developments tend to accelerate
the convergence of iterative methods by modify-
ing the convergence engine with no major modi-
fication of the interset distance cost engine. 19,2~
Although the speed of convergence is certainly
important for widespread acceptance and usage
of the MM concepts, it is important to stress
that it is the cost engine itself that makes the
true identity of a registration approach.
In the scope of functional data acquisition
and imaging of the brain, electromagnetic to-
mography (EMT), either based on electroen-
cephalography (EEG) or magnetoencephalogra-
phy (MEG), can be seen asa functional modality
able to complement PET-SPECT and fMRI
ideally. Compared with them, EMT, which does
not necessarily require complex external de-
vices (eg, when based on EEG), provides the
actual capacity to follow--on an arbitrary time
scale close to real time--the electrical changes
caused in the brain by an external stimulus oran
internal (dys)function. Besides the obvious ad-
vantages of EEG-based EMT (fast, relatively
inexpensive, and no special requirements for
acquisition set-up), this technique generally suf-
fers from its intrinsic low spatial resolution. As
such, it requires accurate matching and fusion
with actual brain anatomy to be correctly inter-
preted and show its full potential for functional
analysis.
Once properly registered with anatomy, the
timing capabilities of EMT should eventually
allow the anatomic path of a (dys)functional
electromagnetic (EM) activity to be traced in
near real time. In addition to the spatial localiza-
tion aspect, and because of the totally new
information they provide, registering the EMT
maps to other modalities can also lead to a
better understanding and assessment of the
underlying functional and metabolic behavior
of the tissues.
Most present modalities being tomographies,
the multisensor space initially has three dimen-
sions. For dynamic acquisitions, time adds a
fourth dimension, and considering various mo-
dalities or sensors adds another one. The whole
MS space ends up being a 5D one (Fig 1), which
does not consider the possible postprocessing
issues (pharmacokinetic modeling in PET, etc),
which can also add parameter dimensions.
Computed
Tomography
(3D)
Angiography
,z
Magnetic
Resonance
Imaging
(Angiography)
(3D)
functional MR
(3D)
MR Spectroscopy
(0 of 3D)
II
EEG (MEG)
(0D+time, lp
2.5D+tirne or ItE
3D+time) H
Electro ]1
MagnetLc II
Tomography II
(3D+time)
Depth Electrodes Ii
(invasive) ti
Z- L__
Single
Photon
Computed
Emission
Tomography
(3D)
various tracers
morphology function - metabolism
Positron
Emission
Tomography
(3D)
various tracers
dynamic
quantification
models
tomography
-> 3D
J /"
............................................................................
modality - sensor= 5thD
(processing -> parameters = 6thD)
5D MultiSensor
space
Fig 1. 5D multisensor space.
THREE- TO FIVE-DIMENStONAL MULTISENSOR IMAGtNG 187
The purpose of this report is to present the
current multisensor environment and means
that have been implemented in Geneva to
register the different data sets and visualize
them simultaneously. Compared with other ap-
proaches, the Geneva integrated multisensor
imaging and processing system (IMIPS) has
been developed to accommodate both clinical
and research protocols and applications. EMT
and EEG have also been integrated in IMIPS as
fuIly featured modalities and can readily be
compared with the other modalities both in
space and time. Because of its rather new entry
in the multisensor field, and because of the
specificity ir adds to IMIPS compared with
other multisensor approaches, there will be
some emphasis on the specific processing and
use of EMT data within this context.
Typical examples of current presurgical assess-
ment of pharmacoresistant epilepsy will be
succinctly presented to illustrate the use and
impact of IMIPS on actual clinicaI cases. For
this specific pathology, better and more objec-
tive localization of the foci is obtained from MS
approaches before possible surgical resection:
the MR data can show possible morphological
lesions and the interictal fluorodeoxy-Glucose
(FDG)-PET shouId show localized hypo-
metabolism22,23; the comparison between interic-
tal and ictal hexamethyl-propylene-amine-
oxime (HMPAO) or ethylene-cysteinate-dimer
(ECD)-SPECT should show an increase of
perfusion, 23,24 and the EEG reconstructed dur-
ing the ictus can show the EM evolution of the
crisis in near real time. Most often, only a few of
the individual modality clues are simultaneously
visible for any single patient. The combination
of data of various types is then expected to
improve the localization of the triggers or poten-
tial epileptic foci from complementary view-
points, and is, therefore, of great interest for the
planning of possible surgical resection or fur-
ther investigation, such as depth electrode im-
plantation.
MATERIALS AND METHODS
Acquisition and Processing Environments
The medical imaging equipment available at the Geneva
Canton University Hospitals (HUG) is a typical multiven-
dor configuration.
MR: Picker Edge 1.5T
(Picker International, Inc, Highland Heights, OH)
CT: Picker PQ2000 spiral CT
(Picker International, Inc, Highland Heights, OH)
SPECT: Toshiba three-head prism
(Toshiba Medical Systems, ]nc, Tustin, CA)
PET: CTI
(CTI, Inc, Knoxville, TN)
Siemens RPT (rotating positron tomograph)
ART (advanced rotating tomograph) rotating PET
(Siemens Medical Systems, Inc, Iselin, NJ)
The EEG acquisition is performed on commercial personal
computer-based multichannel equipment. From the scalp
electrodes' locations and EEG measurements, the current
density vector field can be reconstructed in 3D and low
resolution. For example, Iow resolution electromagnetic
tomography (LORETA), 2s which is used in Geneva, recon-
structs the current density vector field (see Fig 4) by making
various assumptions about the shape of the head (virtual
sphere on which the electrodes lie), the conductivity inside
the head (concentric brain, skull, and scalp spherical shells),
and the continuity of the neuronal generators distribution
within the brain.
All imaging devices ate connected to a hospital-wide
picture archiving and communication system (PACS). 26 At
the present time, archiving of images is performed on a
routine basis only for data from MR and CT equipment,
whereas SPECT and PET data are converted and trans-
ferred only on demand.
Data retrieval, conversion, alignment, fusion, and real-
time 3D visualization ate all performed on a Silicon Graph-
ics (Mountain View, CA) workstation (Crimson Reality
Engine), mostly under Explorer (SGI-Numerical Algo-
rithms Group, OxŸ UK), a module-based multiplatform
software engineering system for which many specific mod-
ules had to be developed to fulfill the needs of multidimen-
sional and MS biomedical imaging.
To comply with the requirements of the PACS, and also
to make IMIPS truly compatible with clinical applications,
all reference and resliced data sets produced in the scope of
IMIPS are converted to the digital imaging and communica-
tion in medicine-compatible 27 Papyrus file format and made
globally available on the PACS for reviewing and processing
on any workstation. With the reviewing software Osiris 2s
available as the standard HUG-PACS display environment,
the registered data sets can then be visualized in side-by-
side or screen-mesh fusion modes, and regions of interest or
other primitives can be drawn on one data set and projected
onto the others for calculations or localization.
Besides the Papyrus files, all the objects created for
display on the MM workstations can also be saved as images
oras 3D objects under the OpenInventor 29 or virtual reality
modeling language VRML ~o standards. These objects can
then be transferred and used for illustration or navigation
through other commerical or public domain multiplatform
software.
Hybrid Registration of Multisensor Data Sets
IMIPS (Fig 2) was developed in Geneva primarily to
allow for the retrieval, registration, and 3D to 5D fusion of
188 BIDAUT ET AL
SPECT-InterFile-]
PET-Matrix_.J
any data format
(text or binary)
MultiDimensionaI-MultiSensor
biomedical imaging
software environment
PACS-Papyrus
Analyze
AIR
(Woods et al.),
SPM
(Friston et al.),
.oo
I
Hat-Head
i
Osiris
(Pelizzari et al.) |
~ MultiSensor
"~
ultiDimensional I
WorkStation
J
Import
MD-MS alignment & processing
(fusion/mapping, segmentation,...)
Export
curves, contours
multimedia, ~ movies
WWW/Internet .... images
A geometries Inventor other 3D packages
-~~'VRML ~ WWW/Internet
PACS/arr
--~l~~reviewing and post-processing workstations.~ <
.,,
I va, .... ,,ae,~~Ÿ
CT, PEl', SPECT markers __ markers alignment:
direr LS, SVD
segmentation:
(ma~or d~screpanr
brain or Scalp, ~ ,~ +
inrtialization of other alignmenl)
__ etc. ~, ~ ] initialization
volume slices
markers ~~
\
L reference MRI segmentation: 9
brain or scalp,
etc.
secundary reference
dataset dataset
jm_ a_g_/ng_ mo_ da_t lt_ ~s_ ................. ~ .....
Non-lmaglng rnodalitiea
[
Fnode Iocations |
i~..~ and EMT valuas " /
I EMT I markers//fiducials /
B
Lelectrodes' Iocations
semi-automated alignment
( Pelizzad, Charolar
or Waods'
optimal transforma#on
I
MultiSensor
alignmant
syatem & protocola
direct, q SVD
manual or
semi-automated alignment
(Pelizzad or Chamfer)
Fig 2. (A) Multidimensional-
multisensor environment and
software for the study of brain
anatomy, metabolism and func-
tion. (B) The whole system and
protocols are all grouped within
IMIPS.
morphological data, such as bone and soft tissue anatomy
extracted from CT, brain anatomy extracted from MR],
angiography data from MR (MRA) or CT (CTA), and
functional data modalities, such as PET, SPECT, or EEG/
MEG-based EMT.
As stated in the introduction, there is no general solution
to registering two data sets, and several established tech-
niques of complementary types were implemented in IMIPS
so that the investigators would be able to salve most
registration prob]ems by optimally combining these tech-
niques with their data.
Whenever possible, a semiprospective protocol is used
THREE- TO FIVE-DIMENSIONAL MULTISENSOR IMAGING 189
that initializes the registration of the more traditional
imaging modalities (MR, CT, PE]', and SPECT) by a
geometric transformation, making use of commercial exter-
nal MM markers (IZI, Baltimore, MD). The markers are
used as they come for CT and MR (they behave like skin),
but require a small dose of radiotracer ( = 0.2 mL of a = 30
~xCi/mL solution) for SPECT and PET acquisitions. They
are placed at the nasion and preauricular cavities for every
acquisition. These locations have been extensively used as
references in many MM projects, 31 as they are easy to
identify and do not more significantly during or between
examinalions. Currently, the authors also use a fourth
"orientation" marker on the right temporal cavity for
solving the possible orientation discrepancies (eg, resulting
from different equipment of protocols) between the various
modalities being considere&
The markers are clearly visible in eve~ data set, and their
location can easily be extracted by interactive pointing on
the reconstructed images. From these Iocations, the geomet-
ric transformation from one set of markers to another one
can be calculated either directly (with only three markers to
consider) or through techniques that can be extended to use
more landmarks. L~a4
From the "gross" initial marker-based alignment step,
the alignment transformation can be refined either manu-
ally of semiautomaticalty. The manual atignment tech-
niques, through interactive 3D surface manipulation or
direct transformation input (le, 3D translations and rota-
tions in the markers' or initiat volume's referential), are still
the standard against which al1 other techniques are com-
pared.~2,32 Even with the help of the marker-based initializa-
tion step and reasonable interaction speed, they can be
tedious to use. For this reason, and also to rely less on the
possibly exclusive expertise of a few users for the bulk of
routine cases, other more automated means of objectively--
and iteratively--refining the alignment between modalities
have also been implemented in IMIPS including 3D surface
approaches, 16 3D (Chamfer) distance maps] 7Js or data
similarity 2~ approaches. These individual techniques, whicb
often imply a preliminary segmentation step, are alt virtually
integrated in a hybrid global system (similar to Bidaut et
aP3), where they are used to solve the specific alignment
steps on which they perform best, to complement of support
the other techniques. Figure 2B explains better how the
various components of IMIPS interact to register specific
MS pairs.
Whenever the original data sets contain similar enough
information (eg, anatomic clues), the quality of the align-
ment can be assessed at anytime by visualizing overlapping
slices or 3D surfaces of similar structures (Fig 3),
Registration of EMT to MR. The standard Tl-weighted
3D MRI acquisition used as spatial reference is performed
on the patient at some time close to the EEG. Before the
EEG recording, all electrode locations ate digitized with a
3D digitizer, such asa Polhemus magnetic system (Pol-
hemus Laboratories, Inc, Colchester, VT) or a passive
robotic arm (Fig 4A). To simplify the alignment step, more
anatomic or fiducial landmarks (such as the markers intro-
Fig 3. Verification of multimodality alignment: surface based (A, brain surfaces from four SPECT data sets} and slice based (B, MR
and SPECT data sets).
190 BIDAUT ET AL
Fig 4. (AI
MR
scalp and EEG electrodes. (81
EMT
spherical
reconstruction
nodes.
(e)
EMT
nodes matched
to
the
actual brain
anatomy. (01
Matched
EMT
vector
field. (E)
Matched
EMT
maximum
current
envelope.
duced previously), which can easily be identified on the
reference MRI
data
set, can also be digitized at this stage.
It
is worth mentioning that the quality of the
EMT-MR
alignment is
dependent
on the quality of this measurement.
As the reconstructed
EMT
maps in their ideal spherical
virtual space-' (Fig 4B) do not contain enough information
about the underlying anatomy, the fiducials and electrodes'
locations on the scalp recorded before the
EEG
(Fig 4A)
are the only information allowing one to register the
EMT
maps to the actual MR anatomy. Because of the presently
low
number
of electrodes ( ""27 in current long-term moni-
toring settings), the alignment is performed
either
manually
from the initial transformation provided by the markers'
locations (ie, by 3D rotations and translations) or with the
help of an iterative search (similar to Pelizzari et aP6 or
Mangin et aP8), which minimizes the 3D distance between
the electrodes'
"cloud"
and the scalp segmented from the
reference MR
data
set.
In addition to the
EMT-MR
alignment, the electrodes'
locations (Fig 4A) are also used to resample spatially, in the
real 3D space where they were initially located, the
EMT
data
reconstructed inside their perfect virtual sphere. To do
so, we use the initial transformations
that
map every
electrode
onto
the surface of the virtual
EMT
head
sphere
before the
EMT
reconstruction.P These transformations
are inverted so that the
EMT
reconstruction nodes, which
are uniformly sampled throughout the virtual reconstruc-
tion sphere (Fig 4B), now have actual 3D space coordinates
within the volume enclosed inside the scalp surface as
defined by the electrodes' true locations (Fig 4C and 4D).
Reslicing
of
Multisensor Data Sets
Once
found, the optimal interset geometric transforma-
tions can be applied to the initial
data
sets for reslicing them
individually to match the reference one spatially (eg, MRI).
As the modalities of interest carry different types of informa-
tion, this reslicing can be performed with different modes
likely to best preserve this individuality. Using all possible a
priori knowledge about the information to be best preserved
during the reslicing (eg, hyposignal or hypersignal), this
operation can be done by
either
nearest, trilinear interpola-
tion or
standard
ranking (minimum, median, or maximum)
filters>' on the trilinear kernel. To improve the appearance
of the resliced
data
sets, anisotropic
data
may also be
interpolated before the reslicing itself by linear interpola-
tion or ranking filtering.
Reslicing
of
the
EMT
data.
After
registration of the
fiducials' or
EEG
electrodes' locations between the MR and
EMT
data
sets (Fig 4A), the four-dimensional (4D) EM
data (true spatial 3D and time =4D) are transformed
through the optimal geometric alignment transformation
into the MRI referential. Because of the low resolution of
the
EMT
reconstruction, the reslicing can take several
forms, depending on the information to be emphasized:
nearest node value, trilinear interpolation, or ranking (mini-
mum, median, or maximum) filters between closest adjacent
nodes. Because of the low resolution and coarse or non-
uniform sampling of the
EMT
nodes in the real 3D space,
every reslicing mode comes with a
"capture
sphere" (CS)
option that controls the contribution of every
EMT
node to
the resliced voxels (elementary elements of volume). The
CS makes sure that the
EMT
nodes are considered within
their actual spatial spreading range, which is a function of
both the physical measurement geometry (eg,
EEG)
and the
EMT
reconstruction technique. Setting the CS radius at any
individual
EMT
node to match the resolution of the virtual
EMT
reconstruction at this location ensures that every
EMT
node contributing to a specific spatial reference voxel
will be properly considered in the
EMT
reslicing calcula-
tions.
Multisensor Visualization
Several techniques can be used to visualize or manipulate
the 3D to 5D MS
data
sets resulting from the IMIPS
approaches.v-" Most figures presented in this report dem-
onstrate the capacity of the tools that have been imple-
mented in IMIPS to take care of visualizing in 2D or 3D any
combination of information extracted from a 5D MS vol-
ume.
Directly from the MS space, slices from every sensor can
be
presented
either side by side or in "colorwash" mode, the
latter
being a superposition and transparency fusion of
individually colored slices (see Figs 3, 7 and 8). Because of
the CS concept, the resliced
EMT
images (see Fig 8A)
actually give clues as to how coarse the
EMT
node sampling
is in a specific
area
or how distant a resliced voxel is from
any initial
EMT
node. The resliced
EMT
data show nonover-
lapping circular shapes in the areas of low
EMT
spatial
resolution and in locations distant enough from actual
EMT
nodes (eg, edges of the brain), while giving apparently more
densely packed voxels in
other
areas (eg, inner brain). In the
images of Fig 8A, where the information to preserve was an
EMT
hypersignal, the CS radius was set to the minimum
distance between distinct
EMT
nodes so that the already
low
EMT
resolution could be kept at its highest.
The brain or
other
structures can also be segmented in
IMIPS from the 3D reference
MRI
data
set by manual or
automated morphotopological segmentation (similar to the
THREE- TO FIVE-DIMENSIONAL MULTISENSOR IMAGING 191
one described in Mangin et altS). The latter, which com-
bines ~hresholding and ma~hernatical morphology concepts,
is robust enough for segmenting the whole brain and seldom
requires manual edition of its results. The masks and
surfaces produced at this step can later be used in subse-
quent processing or visualization.
AII 3D to 4D surfaces extracted from the MS space can be
rendered and manipulated in 3D with OpenInventor -29 or
VRML -3~ compatible multiplatform viewers. They can also
be made transparent to better show their relationship with
inner structures or slices from the MS volume. For example,
see Figs 4E, 5, 7B and 8B, which show isocurrent surfaces
extracted from the realigned EMT data. Depending on the
information of interest in the EMT, and also if a complete
time sequence is being investigated, the threshold used to
catculate the isocurrent surface can either be absolute (on a
global 4D sequence) of relative (on the single 3D time
frame under scrutiny). To provide in a 3D mode some
information of a higher dimension, the investigators can
also show a color-coded 3D to 4D projection of any sensor
(eg, metabolism from PET-SPECT or reconstructed EMT
current densities through time) onto the brain structures
extracted from the morphological MR data set (see Figs 5
and 9).
RESULTS
The techniques and protocols implemented
in IMIPS now allow one to retrieve, register,
and integrate on a routine basis many medical
data of a definite complementary nature. Once
they belong to the same 5D space, data from
any multisensor volume can easily be reviewed,
visualized, compared, and used in further pro-
cessing, which makes the IMIPS approach to-
tally compatible with either clinical or research
applications. The modular design of IMIPS
allows easy implementation of any new tech-
nique or component within the global system at
hand, as soon as they become available or
necessary.
Validation of EMT-MR Registration
EMT is the least imaging modality of the ones
currently in use in IMIPS and is a new entry in
the MS field. For these reasons, the correlation
between the 4D EMT maps and the brain areas
known to respond to specific stimuli in normal
subjects has been verified by using evoked
response potential (ERP) data.
The ERP experiments shown in Figure 5 are
better described in Pascuat-Marqui et al2 s The
stimulus retained for illustration is a visual one:
right hemiretina checkerboard (rHRCB). Spe-
cifically, for the rHRCB test, there are peaks in
the electrical measurements at 100 and 136
milliseconds poststimulus. These instants, which
exhibit a spatial inversion in the localization of
the maximum EM activity, are the ones that
have been retained in Fig 5 to demonstrate both
the timing and localization capabilities of the
EMT within the more global multisensor proto-
col, as well as various ways of representing these
data in relation to the underlying anatomy. The
slight nonpredominant frontal activity seen in
the EMT is actually not significant for the
experiment and is principally due to an EMT
artifact caused by the uneven sampling (tar-
geted at occipital/visual cortex regions) of the
electrodes retained for the EMT calculations.
Clinical Examples
In addition to oncology, 37 where MM applica-
tions have already been largely advocated] ,2
many drug-resistant epileptic patients have al-
ready been investigated. The authors' clinical
epilepsy assessment protocol (Fig 6) is primarily
intended for presurgical evatuation of the pa-
tients. It currently includes 3D MRI (standard
Tl-weighted, 1 mm 3 voxels), interictal FDG-
PET, 22,23 interictal SPECT (initially with
HMPAO and now with ECD, which is much
more stable through time), 23,24 long-term EEG
monitoring with video recording, and ictal ECD-
SPECT (injection < 10 minutes after the begin-
ning of a crisis detected by EEG). Other MR
sequences or functional acquisitions (fMRI,
post-ictal SPECT, etc) may also be included
whenever needed or available.
Two typical cases are described briefly in this
report to iltustrate the MS investigation the
patients undergo and the clinical benefits it can
provide.
Patient 1 has a lesion (cavernoma) that can
be easily seen on the MR data set (Fig 7). PET
and SPECT, respectively, show focal hypome-
tabolism and hypoperfusion in the area of the
lesion (Fig 7A). EMT maps at various moments
during the ictus (Fig 7B), and particularly at the
onset, always show an elevated EM activity in
the area of the lesion. Although the PET and
SPECT findings (which are not epilepsy re-
lated) do not permit any conclusion to be
reached regarding the epilepsy source, the EMT
findings point to the functionally involved lesion
as the most probable source of epileptic activity.
192 BIDAUT ET AL
The outcome of this investigation will be a
resection of the lesion, with the help of the MR
spatial reference to localize it better.
Patient 2's MR does not show any visible
lesion or tissue damage. Ictal SPECT and EMT
both show an area of increased perfusion and
EM activity in the frontal cortex (Fig 8A). The
spatial gap between the two sites seen in Fig 8B
is not truly significant and is further explained in
the Discussion.
For this patient too, other modes of MS
visualization are also shown (Fig 8) with the
projection of ictal SPECT and EMT data on the
outer and inner surface of the cortex extracted
from the MR data set. These modes visualize
both sides of the cortex, and allow for better
assessment of the relationship between the right
frontal SPECT hyperperfusion and the maxi-
mum EM activity during the ictus regarding the
surrounding neurological anatomy (eg, sulci).
Even retrospectively, there are no visible lesions
on the morphological MR data set at the loca-
tion of the functional abnormality. As some EM
activity (while of lesser amplitude) is also mea-
sured at other locations during the ictus, depth
electrodes will be implanted with the help of the
multisensor data. Their electrical measure-
ments will then be used to probe for other
possible locations for the epileptic trigger(s) or
focus (Fig 9).
DISCUSSION
MM registration.
The MM alignment and
reslicing aspects of the IMIPS system described
in this report have been designed to minimize
changes to the current routine acquisition proto-
cols while allowing the use of the data in the
most complex MS protocols. As such, minor
adaptations were necessary to allow for the
proper use of a few markers to resolve orienta-
tion differences between the different data sets
and provide a good initial interset geometric
transformation for subsequent alignment steps.
This basic setting should be kept as long as the
data sets do not carry reliable orientation infor-
mation or are not similar enough across exami-
nations. For example, the use of much different
(and multivendor) tomography equipment with
their own data orientation (ie, left/right, head/
foot) does not always allow for the easy absolute
orientation of the initial data sets. In the near
future, it is hoped, a wider use of medical file
standards 27 should efficiently solve the prob-
lems of spatial compatibility by propagating
appropriate reference and orientation data
structure along with every data set. Also on the
bright side, the lack of significant distortion from
the tomographic imaging modalities (as measured
by standard quality control procedures) obviated
the need for a specific geometry correction comlx>
nent, such as coordinate scaling or warping. 38
Working with only a few markers and also
with low resolution (and often anisotropic)
imaging devices, the geometric transformation
extracted from the markers alone does not
allow for perfect alignment of the data sets.
Nevertheless, the initial result takes care of
the orientation differences and is near the
ideal solution. This proximity then helps itera-
Fig 5. Visual ERP (right hemiretina checkerboard) EMT at
two selected instants. EEG potentials recorded during a rHRCB
ERP exhibit two peaks at 100 and 136 milliseconds (see EEG
curves). The EMT reconstructions at these two moments show the
characteristic inversion of the maximum EM activity from the right
side (100 mil|iseconds) to the left side (136 milliseconds) through
the display of the maximum current envelope. The other display
modes al|ow for better assessment of the link between the EM
activity and the underlying anatomy (outer and inner cortex).
Although only two characteristic time frames have been selected
for
this figure, each single time frame (evew millisecond for an
ERP) can be projected in the same way to follow dynamically the
evolution of the EM signal through time and in relation with the
anatomy.
Fig 7. Patient 1: (A) Multimodality slices showing PET
hypometabolism and SPECT hypoperfusion at the Iocation of
the les|on seen on the MR slice. (B) Four different moments
following the onset of an ictus; the EMT current during the
discharge always stays at an elevated value (as shown by the
EMT maximum current envelope) in the area of the les|on (at
the accuracy of EMT).
Fig 8. Patient 2: (A) Multimodality slices; the s|ices exhibit
an ictal SPECT hyperperfusion (middle columP.) and an ictal
EMT hypercurrent (right co|umn). The appearance of the EMT
slices is further described in the article. (B) Comparison of the
SPECT and EMT data, and the Iocations of maximum EMT
current and maximum SPECT perfusion.
Fig 9. Patient 2: Projection of functional data (EMT
[top
row] and ECD-SPECT
[Iower
row]) on the patient's brain
segmented from the MR data set. The left column shows the
projection of the functional data on the outer surface of the
cortex. The right column shows the inner surface of the cortex
(= outer surface of the white matter), which is more closely
related to the sulci and shows a different side of the relation
between functional data and underlying anatomy. For this
patient, the interest of the mapping is to show the relation
between the anatomy and maximum EM activity, and meta-
bolic perfusion seen in the right frontal cortex during the ictus.
Figure 5
Figure 7
Figure 8
194 BIDAUT ET AL
EEG scalp electrodos
~MEG sensors J
EEG-MEG acquisition
1
~ multichannel analyzer
+ video recording
Ir .... ,ou.,m.,/f_g/
IE.T ..... ,,uo,ioo I
tracer
c us*
i hemisphedCal refetential
,o,oc,,on~ s~
-<- ..................
J
A
MD-MS
environment & protocol
for epilepsy (*)
and brain function assessment
fMRI, MRS .....
[
~~
reviewing
workstations
surface EEG
<-> EMT
Mil
inte¡ PET
o
B
Pre-surgical epilepsy assessment
protocol
inconsistent
..................... i
i
~iW_aŸ interictal SPEC~ ]
91
....... I_ ......... 2 .... 1
ve tests
ir
invasive EEG r
)_ _ _ _ i _ ~_~_~ ~ ~!%Ÿ237 _ i
surgery y
Fig 6. Epilepsy protocol environment (A) and decision tree
(B). The asterisks in Fig 6A mark the steps required for the
presurgical assessment of epilepsy.
tive rtiethods to avoid local minimas in which
they could be trapped--far from the actual
solution.
The IMIPS multicomponent alignment ap-
proach was built on published and recognized
developments, the accuracy of which is strongly
dependent on the resolutions and types of the
modality pairs under scrutiny. By using a prelimi-
nary marker step, a final possible manual inter-
action, a best-choice method, and state-of-the-
art acquisition devices (which were not always
available at the time of the initial develop-
ments), registration solutions should obtain an
accuracy similar or better than the original
published values, la3z Still, andas actually men-
tioned in the original publications, 12,32 align-
ment results occasionally require visual check-
ing and final manual correction, whenever
enough similar clues are visible on both data
sets (which is the case with the sequences and
tracers we are currently using). This "poor
performance" is generally due to spatial discrep-
ancies between the incoming data sets (eg, too
much anisotropy) or to too many differences in
the interset features used by the alignment
methods (eg, incomplete brain coverage). When
not specifically linked to intrinsic limitations,
such as tracer choice in PET or SPECT, these
differences can sometimes be anticipated and
corrected at the initial acquisition/reconstruc-
tion time by clearly stating before the individual
examinations which ones are planned to be used
in a MS protocol.
EMT-MR registration.
The EMT recon-
structed data are of low resolution; in its pre-
sent setting, LORETA can resolve two point
sources if separated by more than 25 mm. With
this intrinsic limitation, the simple nasion and
preauricular fiducials' alignment of an EMT
map with the MR spatial reference generally
gives adequate results. Even so, manual interac-
tion can stilt be necessary to correct for possible
problems, such as patient motion during digitiz-
ing of the electrodes' locations. The limited
number of electrodes in the current routine
EEG recording protocol (= 27) does not allow
much better results to be reached when itera-
tively minimizing, by techniques, such as those
used by Pelizzari et al 16 or Mangin et at, 18 the
distance between the electrodes' locations and
the relatively smooth 3D scalp surface extracted
from the MR data set. If, as it is planned, the
number of electrodes is increased from 27 to 64
or even I28, the more automated iterative align-
ment techniques might give better results, compat-
ible with a higher-resolution EMT reconstruction.
THREE- TO FIVE-DIMENSlONAL MULTISENSOR IMAGING 195
Reslicing.
The ranking filters reslicing inter-
polation modes are an attempt to make interpo-
lation a little smarter without going to the
expense of full shape-based interpolation, 39
which would require preliminary segmentation
of the data sets. Although the latter interpola-
tion technique certainly produces better results,
at least near the edges of the structures of
interest, segmentation of the initial shapes is
not always possible based on only single modal-
ity data sets. Besides, one should always remem-
ber that interpolation does not objectively im-
prove the initial content of the data. Its major
goal is attempting to .re-create, under reason-
able assumptions, what is missing or has been
lost at previous steps.
Visualization.
By going further in the seg-
mentation of the brain by windowing MR signal
values or going to more statistical ap-
proaches, 4~ the values of the nearby EM or
metabolic activity can be mapped onto the outer
and inner surfaces of the cortex (Figs 5 and 9) or
even more specific structures within the brain.
As it stands, the ability to peel away the cortex
(from outer cortex to outer white matter) and
project on these anatomic structures data from
other sensors is expected to improve the under-
standing of brain function. Sulci segmentation
and identification 42 will eventually further this
approach.
Validation and clinical examples.
Investiga-
tions, including depth electrodes EEG, are stil|
being conducted to assess the correlation be-
tween structure and function (by PET, SPECT,
and other EEG methods as well as by EMT)
better for pathological conditions, such as epi-
lepsy.
Although some patients exhibir morphologi-
cal tesions, which are easily identified on the
routine MRI and likely related to their epilepsy,
others do not present any morphological
anomaly. For these latter patients, the informa-
tion brought in by the functional modalities is
absolutely necessary to further the investiga-
tion. Even if PET alone does not show any loca|
hypometabolism, interictal and ictal SPECT can
still be compared, either qualitatively (eg, by
normalizing the ECD uptake values to the
cerebella hemispheres in both data sets) of
semiquantitatively (eg, by histogram-based inter-
set normalization or by kinetic modeling ap-
proaches). In the two clinical examples pre-
sented in this report, it is interesting to notice
the slightly different type of complementary
information that comes out of the MS approach
for the two types of patients. In cases like
patient 2, preliminary MS results are not conclu-
sive enough, and further investigations need to
be performed (eg, with the help of implanted
depth electrodes or more targeted long-term
scalp EEG monitoring). Also, as qualitative
measurements are not always enough to assess
tissue function properly, 43,44 an effort is under
way to integrate more quantitative procedures
and data (pharmacokinetic modeling approaches
for PET and SPECT, statistics, etc) within the
globa| MS space currently dealt with by IMIPS.
In Fig 8B, the spatial gap between the two
sites of maximal EMT current and SPECT
perfusion is well within the limits of the EMT
resolution accuracy. Also because of noticeably
different intrinsic principles between the two
sensors (particularly their distinct timing capa-
bilities), the sites can reasonably be considered
as spatially related. Moreover, the dynamic in-
formation provided by the "motion" of the EM
maximum activity during the ictus could actually
be used to high|ight specific areas of the brain
with "slower" criteria than instant EM activity.
The new parametric areas might then be com-
pared with MR morphology or function, and
with SPECT or PET (quantified) metabolism.
Intra-patient versus inter-patient.
For the time
being, the IMIPS protocols and system pre-
sented in this report have been designed primar-
ily to accommodate analyses performed on
single individuals. Although IMIPS performs
the task well, there is a definite need to expand
it further to interindividual processing and anaty-
sis for studying neurological function and me-
tabolism in a more global population. Such an
expansion is occurring in two separate direc-
tions to accommodate the diversity and specific-
ity of the investigations. The first direction is to
use an atlas-based approach, such as statistical
parametric mapping, 45 whereas the second ap-
proach tries to address the interindividual ana-
tomic disparities better by using approaches,
such as described in Mangin et al, 42 asa front-
end to more complex statistics.
196 BIDAUT ET AL
EMT related issues. Although LORETA is
currently being used in Geneva for EMT calcu-
lations, other techniques (such as commercially
available time-varying dipole modeling 46) exist
to reconstruct EM data from EEG or MEG
measurements. Although these techniques all
try to address similar issues in attempting to find
"the" best solution to the inverse problem of
localizing the physiological generators from the
physical measurements, they rely on fundamen-
tally different principles. As IMIPS is not lim-
ited to a single EMT method, a direct compari-
son of these methods is beyond the scope of this
article.
In subsequent developments, refinements
should and will be made in some aspects of the
EMT reconstruction, at least to further its
validation in parallel with the other modalities.
These refinements are actually similar to the
ones "already occurring in the other functional
and metabolic imaging fields where CT and MR
data are expected to improve the reconstruction
and processing of PET, SPECT, and fMRI data.
The "capture sphere" concept developed in
IMIPS for the reslicing of EMT data is evolving
toward a more realistic shape that should better
consider the actual non-uniform spatial sam-
pling and spreading of the EMT nodes in all
three dimensions. The EMT coarse sampling is
both due to the nonuniform electrodes' spacing
and to the EMT reconstruction assumptions. As
currently done for ERP experiments, the elec-
trodes' locations can be tuned to cover better
specific locations, such as the temporal lobes or
other areas under scrutiny. From this locally
improved spatial sampling at the physical mea-
surement level, the EMT calculations can then
also provide a better spatial sampling at the
reconstruction level in the selected areas. As
the EMT investigation for epilepsy is currently
being performed in parallel with the routine
long-term neurological investigation, the num-
ber of electrodes is relatively low and does not
evenly cover the whole brain. In such a setting,
LORETA, which can make use of any addi-
tional information, such as depth electrode
measurements, still reconstructs, although with
low resolution, all sources within the volume
and can discriminate between individual maxi-
mums if they are at least separated by approxi-
mately 25 mm. Modifying the number of elec-
trodes or tuning the electrodes' spatial sampling
to more specific areas would modify this discrimi-
nation distance.
CONCLUSION
The open and modular MS facility (IMIPS)
has been developed and implemented in Geneva
to take care of all needs for both clinic- and
research-oriented MS imaging and data process-
ing.
Besides efficiently addressing the clinically
compatible registration issue for various sensor
pairs, IMIPS also includes some powerful visual-
ization concepts and tools to present the infor-
mation from a 5D MS space in a way understand-
able by both scientists and nontechnically
oriented physicians.
The authors have shown the feasibility and
potential interest of including--through means
easily implemented in the clinical routine--
EEG(-MEG) EMT in an already complex and
rich (although more traditional) MM space that
includes MR-CT anatomy and PET-SPECT
metabolism. Besides FDG-PET and SPECT,
which are now performed on a routine basis, 3D
echo-planar fMRI and MRS should also be
soon available in Geneva, along with better
access to more marked PET molecules pro-
duced in a new radiochemistry laboratory. In its
present shape, but even more so by including
MEG, IMIPS can already be extensively used
for studying normal and pathological brain me-
tabolism and function, for example, through
ERP's and other functional modalities (PET or
fMRI).
In Geneva, the authors decided to target
initially only a limited number of protocols with
an expected added value ffom MS approaches.
The MS grading and therapy planning of tumors
and the assessment of pharmacoresistant epi-
lepsy are such protocols. In addition to provid-
ing an improved opportunity to gain a better
knowledge of the underlying in vivo metabolism
and (dys)function of the tissues, the anatomi-
cally related multisensor space should also al-
low for better planning and monitoring of any
subsequent therapy. Further protocols will be
introduced as the authors learn better from the
ones on which they initially focused.
THREE- TO FIVE-DIMENSIONAL MULTISENSOR IMAGING 197
When considering all the functional sensors
now at hand, the total richness of the resulting
combined MS space has not yet been fully
appreciated at the moment. The newly envi-
sioned MS potential, primarily based on mul-
tidisciplinary approaches, should never hide the
complexity of putting together data (and people)
from many scientific and technical fields, even
for rather targeted research protocols. Only by
properly assessing the means and educating the
potential users will prevention of the disappoint-
ments that might come from less reasonable
expectations be possible.
ACKNOWLEDGMENT
The authors thank Prof D. Lehmann for providing the
EEG-ERP recordings initially used for EMT-MR valida-
tion.
REFERENCES
1. Wahl RL, Quint LE, Cieslak RD, et al: Anatometa-
bolic tumor imaging: Fusion of FDG-PET with CT or MRI
to localize foci of increased activity. J Nucl Med 34:1190-
1197, 1993
2. Pirotte B, Goldman S, Bidaut LM, et al: Use of
positron emission tomography (PET) in stereotactic condi-
tions for brain biopsy. Acta Neurochir 134:79-82, 1995
3. Pomeranz S J, Pretorius HT, Ramsingh PS: Bone
scintigraphy and multimodality imaging in bone meoplasia:
Strategies for imaging in the new health care climate. Semin
Nucl Med 24:188-207, 1994
4. Freitas JE, Freitas AE: Thyroid and parathyroid
imaging. Semin Nucl Med 24:234-245, 1994
5. Alexander C, Villena-Heinsen CE, Backens M, et al:
Image fusion of MRI and immunoscintigraphy with MAb-
170 in ovarian tumors. Nuklearmedizin 34:173-178, 1995
6. Fox PT, WoldorffMG: Integrating human brain maps.
Curr Opin Neurobiol 4:151-156, 1994
7. Tamraz J: Neuroradiologic investigation of the visual
system using magnetic resonance imaging. J Clin Neurophys
11:500-518, 1994
8. Brunetti A, Soricelli A, Rotondo A, et al: Correlative
morphologic and functional imaging for diagnosis, staging
and follow up in AIDS: Ah overview. Q J Nucl Med
39:201-211, 1995
9. Elgazzar AH, Abdel-Dayem HM, Clark JD, et al:
Multimodality imaging of osteomyelitis. Eur J Nucl Med
22:1043-1063, 1995
10. Ratib O, Didier D, Chatelain P, et al: Standard views
in cardiac multimodality tomographic imaging. Ana J Card
Imaging 9:67-76, 1995
11. Ferris JV, Marsh JW, Little AF: Presurgical evalua-
tion of the liver transplant candidate. Radiol Clin North Am
33:497-520, 1995
12. Van den Elsen P, Pol E-JD, Viergever MA: Medical
image matching: A review with classification. IEEE Eng
Med Biol 12:26-38, 1993
13. Arun KS, Huang TS, Blostein SD: Least square
fitting of two 3D point sets. IEEE Trans Pattern Mach Intell
9:698-700, 1987
14. Umeyama S: Least square estimation of transforma-
tion parameters between two point patterns. IEEE Trans
Pattern Mach Intell 13:376-380, 1991
15. Dhawan AP, Arata LK, Levy AV, et al: Iterative
principal axes registration method for analysis of MR-PET
brain images. IEEE Trans Biomed Eng 42:1079-1087, 1995
16. Pelizzari CA, Chen GTY, Spelbring DR, et al:
Accurate three-dimensional registration of CT, PET, and/or
MR images of the brain. J Comput Assist Tomogr 13:20-26,
1989
17. van Herk M, Kooy HM: Automatic three-dimen-
sional correlation of CT-CT, CT-MRI, and CT-SPECT
using chamfer matching. Med Phys 21:1163-1178, 1994
18. Mangin J-F, Frouin V, Bloch I, et al: Fast nonsuper-
vised 3D registration of PET and MR images of the brain. J
Cereb Blood Flow Metab 14:749-762, 1994
19. Woods R, Mazziotta JC, Cherry S: MRI-PET registra-
tion with automated algorithm. J Comput Assist Tomogr
17:536-546, 1993
20. Meyer CR, Leichtman GS, Brunberg JA, et al:
Simultaneous usage of homologous points, lines and planes
for optimal 3D linear registration of multimodality imaging
data. IEEE Trans Med Imaging 14:1-11, 1995
21. Alpert NM, Berdichevsky D, Levin Z, et al: Improved
methods for image registration. Neuroimage 3:10-18, 1996
22. Gaillard WD, White S, Malow B, et al: FDG-PET in
children and adolescents with partial seizures: Role in
epilepsy surgery evaluation. Epilepsy Res 20:77-84, 1995
23. Nagata T, Tanaka F, Yonekura Y, et al: Limited
value of interictal brainperfusion SPECT for detection of
epileptic foci: High resolution SPECT studies in compari-
son with FDG-PET. Ann Nucl Med 9:59-63, 1995
24. Matsuda H, Yagishita A, Tsuji S, et al: A quantitative
approach to technetium-99m ethyl cysteinate dimer: A
comparison with technetium-99m hexamethylpropylene
amine oxime. Eur J Nucl Med 22:633-637, 1995
25. Pascual-Marqui RD, Michel CM, Lehmann D: Low
resolution electromagnetic tomography: a new method for
localizing electrical activity in the brain. Int J Psychophysiol-
ogy 18:49-65, 1994
26. Ratib O, Ligier Y, Scherrer J-R: Digital image
management in medicine. Comput Med Imaging Graph
18:73-84, 1994
27. Bigwood WD, Horii S: Introduction to the ACR-
NEMA DICOM standard. Radiographics 12:345-355, 1992
28. Ligier Y, Ratib O, Logean M, et al: Osiris: A medical
image manipulation system. MD Computing Journal 11:212-
218, 1994
29. Wernecke J: The Inventor Mentor. Reading, MA,
Addison-Wesley, 1994
30. Ames AL, Nadeau DR, Moreland JL: The VRML
Source Book. New York, NY, Wiley, 1996
31. Gleason PL, Kikinis R, Altobelli D, et al: Video
198 BIDAUT ET AL
registration virtual reality for non linkage stereotactic sur-
gery. Stereotact Funct Neurosurg 63:139-143, 1994
32. Turkington G, Jaszczak RJ, Pelizzari CA, et al:
Accuracy of registration of PET, SPECT and MR images of
brain phantom. J Nucl Med 34:1587-1594, 1993
33. B[daut LM: Composite PET and MR[ for accurate
localization and metabolic modeling: A very usefu[ too[ for
research and clinic. SPIE Medical Imaging V 1445:66-77,
1991
34. Oonzalez RC, Wintz P: Digital Image Processing.
Reading, MA, Addison-Wesley, 1987
35. Barillot C: Surface and volume rendering techniques
to display 3D data. 1EEE Engineering in Medicine and
Biology 12:111-119, 1993
36. Toga A (ed): Three-Dimensional Neuroimaging. New
York, NY, Rayen, 1990
37. Bidaut LM, Levivier M, Goldman S, et al: Multimo-
dality stereotaxic correlation between XCT, PET, MRI and
histology for tumoral tissue evaluation in the brain_ Proceed-
ings of the 14th Annual International Conference of the
IEEE-EMBS 1215-1216, 1992
38. Bookstein FL: Principal warps: Thin-plate splines
and the decomposition of deformations. IEEE Trans Pat-
tern Anal Mach Intell 11:567-585, 1989
39. Beier T, Neely S: Feature-based image metamorpho-
sis. Comput Graph 26:35-40, 1992
40. Liang Z: Tissue classification and segmentation of
MR images. IEEE Eng Med Biol 12:81-85, 1993
41. Wells WM, Grimson WEL, Kikinis R, et al: Adapta-
tive segmentation of MRI data. Ayache N (ed): Computer
Vision, Virtual Reality and Robolics in Medieine. New
York, NY, Springer-Verlag, 1995, pp 59-69
42. Mangin J-F, Frouin V, Bloch I, et al: Automatic
construction of ah attributed relational graph representing
the cortex topography using homotopic transformations.
SPIE Mathematical Methods in Medical Imaging III 2299:
110-121, 1994
43. Sperling MR, Alavi A, Reivich M, et al: False
lateralization of temporal lobe epilepsy with FDG positron
emission tomography. Epilepsia 36:722-727, 1995
44. Gaillard WD, Fazilat S, White S, et al: Interictal
metabolism and blood flow are uncoupled in temporal lobe
cortex of patients with complex partial epilepsy. Neurology
45:1841-1847, 1995
45. Friston K J, Frith CD, Liddle PF, et al: Comparing
functional (PET) images: The assessment of significant
change. J Cereb Blood Flow Metab 11:690-699, 1991
46. Verkindt C, Bertrand O, Perrin F, et al: Tonotopic
organization of the human auditory cortex: N100 topogra-
phy and multiple dipole model analysis. Electroencephalogr
Clin Neurophysiol 96:143-156, 1995
New Digital Imaging Book Announcements
Picture Archiving and Communication Systems in Biomedical lmaging
by H.K. Huang, DSc
VCH Publishcrs, Inc.; 303 NW 12th Avenue; Deerfield Beach, FL 33442; Fax:
800-367-8247 Outside of North America, VCH Verlagsgesellschaft mbH; P.O.
10-11-61; 69451 Weinheim FRG.
$150.00 plus handling. Call 800-961-8249.489 pages.
lmaging
and Information Management:
Computer Systems for a
Changing
Health Care Environment (May, 1996 Radiological Clinics of North ~America)
by Robert A. Greenes, MD, PhD, and Roger A. Bauman, MD
W.B. Saunders Co.; Pcriodicals Marketing; Independence Square West; Philadel-
phia, PA 19106; Fax: 407-363-9661
$31.00 (includes handling); add $6.00 for airmail to Europe)
Call 800-654-2452. 248 pages.
... The decisive information does not often appear on every one (if any) of the data sets, but the combining of all data-including the more traditional neuropsychology investigation results-generally permits the most likely focus to be pointed at with enough accuracy for deciding on subsequent therapy (Bidaut, in press;Bidaut et al., 1996;Duncan, 1997;Knowlton et al., 1997;Levin et al., 1989). Among all the imaging modalities previously mentioned, only MRI and CT provide the anatomical clues mandatory to the planning or monitoring of surgery. ...
... Handling techniques and data as varied as the ones mentioned previously in a common environment eventually led to the integration of all software and imaging protocol elements within a virtual hybrid system that was dubbed Integrated Multidimensional and Multisensor Imaging and Processing System (IMMIPS). Because some of IMMIPS's functions, techniques, and means have already been described before (Bidaut et al., 1996), they will only be summarized herein for the sake of completeness. ...
... Segmentation techniques extract structures of interest from a volume of data either for visualization or for subsequent processing. These techniques generally range from direct manual editing/tagging of the data-which can now be easily performed on many commercial workstations-to (semi-)automatic morphology-based (Bidaut et al., 1996;Mangin, Frouin, Bloch, Bendriem, & Lopez-Krahe, 1994), statistical (Held et al., 1997;Nocera & Gee, 1997;Wells, Grimson, Kikinis, & Jolesz, 1996), or even more advanced operators that attempt to further refine and streamline these tedious operations for addressing specific or more general needs. ...
Article
Full-text available
Images and data representing morphology, function, and metabolism are medically produced in the scope of specific protocols and pathologies. Although generally assessed individually, these data can also be merged together into a multidimensional digital space that simulates the reality of the patient from many different perspectives. This space can then be navigated at will along any dimension, not only for diagnosis but also to plan, simulate, or monitor surgery or other types of therapy. These approaches can be used for addressing various research or clinical needs such as the primary and secondary assessments of complex pathologies, including cancer and epilepsy. Concepts such as those presented in this article represent an adequate foundation not only for an optimum exploitation of all available information for in-vivo assessment and therapy guidance but also for a more effective assessment of the full scope and requirements of live systems simulation(s) accurate enough for supporting clinical and research developments.
... A similar process was used to register MR to CT, albeit with a high-resolution FOV that was limited to anatomy that was visible within the physical tumor window. Once registered together, the PET, CT, and MR data sets were fused together, processed, and visualized (19) to localize and compare PET uptakes and MR parametric maps in relation to structural anatomy. ...
Chapter
Integrated PET and CT imaging is changing the traditional PET imaging procedures and workflow and raises new technical challenges with regards to image acquisition and data management. The ability to combine morphological and molecular information in a single diagnostic procedure will progressively lead to changes in clinical pathways and patient management. It is therefore important to understand and adapt the underlying technical infrastructure and software tools to support those changes.
Conference Paper
Musculoskeletal disorders (MSD) are becoming a big healthcare economical burden in developed countries with aging population. Classical methods like biopsy or EMG used in clinical practice for muscle assessment are invasive and not accurately sufficient for measurement of impairments of muscular performance. Non-invasive imaging techniques can nowadays provide effective alternatives for static and dynamic assessment of muscle function. In this paper we present work aimed toward the development of a generic data structure for handling n-dimensional metabolic and anatomical data acquired from hybrid PET/MR scanners. Special static and dynamic protocols were developed for assessment of physical and functional images of individual muscles of the lower limb. In an initial stage of the project a manual segmentation of selected muscles was performed on high-resolution 3D static images and subsequently interpolated to full dynamic set of contours from selected 2D dynamic images across different levels of the leg. This results in a full set of 4D data of lower limb muscles at rest and during exercise. These data can further be extended to a 5D data by adding metabolic data obtained from PET images. Our data structure and corresponding image processing extension allows for better evaluation of large volumes of multidimensional imaging data that are acquired and processed to generate dynamic models of the moving lower limb and its muscular function.
Article
Following recent developments, most brain imaging modalities (MR, CT, SPECT, PET) can nowadays be registered and integrated in a manner almost simple enough for routine use. By design though, these modalities are still not able to match the principles and near real-time capabilities of the much simpler (but of lower spatial resolution) EEG, thus the need to integrate it as well, along with - for some patients - the more accurate invasive electrophysiology measurements taken directly in contact with brain structures. A standard control CT (or MR) is routinely performed after the implantation of invasive electrodes. After registration with the other modalities, the initial estimates of the electrodes' locations extracted from the CT (or MR) are iteratively improved by using a geometrical model of the electrodes' arrangement (grids, strips, etc.) And other optional constraints (morphology, etc.). Unlike the direct 3D pointing of each electrode in the surgical suite - which can still act as a complementary approach - this technique estimates the most likely location of the electrodes during monitoring and can also deal with non cortical arrangements (internal strips, depth electrodes, etc.). Although not always applicable to normal volunteers because of its invasive components, this integration further opens the door towards an improved understanding of a very complex biological system.
Article
Introduction: Rapid prototype modeling (RPM) has been used in medicine principally for bones - that are easily extracted from CT data sets - for planning orthopaedic, plastic or maxillo-facial interventions, and/or for designing custom prostheses and implants. Based on newly available technology, highly valuable multimodality approaches can now be applied to RPM, particularly for complex musculo-skeletal (MSK) tumors where multimodality often transcends CT alone. Methods: CT data sets are acquired for primary evaluation of MSK tumors in parallel with other modalities (e.g., MR, PET, SPECT). In our approach, CT is first segmented to provide bony anatomy for RPM and all other data sets are then registered to the CT reference. Parametric information relevant to the tumor's characterization is then extracted from the multimodality space and merged with the CT anatomy to produce a hybrid RPM-ready model. This model - that also accommodates digital multimodality visualization - is then produced on the latest generation of 3D printers, which permits both shapes and colors. Results: Multimodality models of complex MSK tumors have been physically produced on modern RPM equipment. This new approach has been found to be a clear improvement over the previously disconnected physical RPM and digital multimodality visualization. Conclusions: New technical developments keep opening doors to sophisticated medical applications that can directly impact the quality of patient care. Although this early work still deals with bones as base models for RPM, its use to encompass soft tissues is already envisioned for future approaches.
Article
Full-text available
We propose a fully nonsupervised methodology dedicated to the fast registration of positron emission tomography (PET) and magnetic resonance images of the brain. First, discrete representations of the surfaces of interest (head or brain surface) are automatically extracted from both images. Then, a shape-independent surface-matching algorithm gives a rigid body transformation, which allows the transfer of information between both modalities. A three-dimensional (3D) extension of the chamfer-matching principle makes up the core of this surface-matching algorithm. The optimal transformation is inferred from the minimization of a quadratic generalized distance between discrete surfaces, taking into account between-modality differences in the localization of the segmented surfaces. The minimization process is efficiently performed via the precomputation of a 3D distance map. Validation studies using a dedicated brain-shaped phantom have shown that the maximum registration error was of the order of the PET pixel size (2 mm) for the wide variety of tested configurations. The software is routinely used today in a clinical context by the physicians of the Service Hospitalier Frédéric Joliot (>150 registrations performed). The entire registration process requires ~5 min on a conventional workstation.Keywords: Automatic registration; Chamfer matching; Free-form surface matching; Anisotropic distance map; Anatomicofunctional correlation
Article
Full-text available
Two point sets { pi } and { p’i }; i = 1, 2, … … …, N are related by p’i= Rpi+ T + Ni, where R is a rotation matrix, T a translation vector, and Ni a noise vector. Given {pi} and { p’i }, we present an algorithm for finding the least-squares solution of R and T, which is based on the singular value decomposition (SVD) of a 3 × 3 matrix. This new algorithm is compared to two earlier algorithms with respect to computer time requirements. Copyright © 1987 by The Institute of Electrical and Electronics Engineers. Inc.
Article
The decomposition of deformations by principal warps is demonstrated. The method is extended to deal with curving edges between landmarks. This formulation is related to other applications of splines current in computer vision. How they might aid in the extraction of features for analysis, comparison, and diagnosis of biological and medical images is indicated.
Article
In order to help in analyzing PET data and really take advantage of their metabolic content, a system was designed and implemented to align and process data from various medical imaging modalities, particularly (but not only) for brain studies. Although this system is for now mostly used for anatomical localization, multi-modality ROIs and pharmaco-kinetic modeling, more multi-modality protocols will be implemented in the future, not only to help in PET reconstruction data correction and semi-automated ROI definition, but also for helping in improving diagnostic accuracy along with surgery and therapy planning.
Book
This book reports on new neuroimaging technologies that are revolutionizing the study of the brain be enabling investigators to visualize its structure and entire pattern of functional activity in three dimensions. The book provides a theoretical and practical explanation of the new science of creating three-dimensional computer images of the brain. The coverage includes a review of the technology and methodology of neuroimaging, the instrumentation and procedures, issues of quantification, analytic protocols, and descriptions of neuroimaging systems. Examples are given to illustrate the use of three-dimensional enuroimaging to quantitate spatial measurements, perform analysis of autoradiographic and histological studies, and study the relationship between brain structure and function.
Chapter
We propose an algorithm allowing the construction of a high level representation of the cortical topography from a T1-weighted 3D MR image. This representation is an attributed relational graph (ARG) inferred from the 3D skeleton of the object made up of the union of gray matter and cerebro-spinal fluid enclosed in the brain hull. In order to increase the robustness of the skeletonization, topological and regularization constraints are included in the segmentation process using an original method: the homotopically deformable regions. This method is halfway between deformable contour and Markovian segmentation approaches. The 3D skeleton is segmented in simple surfaces (SSs) constituting the ARG nodes (mainly sulcus parts). The ARG relations are of two types: first, the SSs pairs connected in the skeleton; second, the SSs pairs delimiting a gyrus. The described algorithm has been developed in the frame of a project aiming at the automatic detection and recognition of the main cortical sulci. Indeed, the ARG is a synthetic representation of all the information required by the sulcus identification. This project will contribute to the development of new methodologies for the human brain functional mapping.
Conference Paper
This paper presents the design and implementation of a protocol and software system which, after acquisition and transfer, allows the registration and combination of images and data from various and complementary biomedical imaging modalities (X C T , P E T , M R I ). This combination is initially used for locating, inside the referential of a commercial stereotaxic frame, and either on morphological and/or metabolical grounds, the best target contender (i.e. the highest grade tumor tissue) for neurosurgical biopsy. Following initial localization, the system is also used to locate the biopsy samples into the multimodality volume, and for correlating the actual histology findings with the relative and/or modelled (dynamic acquisitions) data values at the samples locations. For now, we mostly rely on XCT and <sup>18</sup>FDG PET for locating the 'metabolical center' of a tumor, later on we will also use other PET tracers and more of MRI data. The system, which deals with file management, multimodality image registration & display, and pharmacokinetic data modeling, should be even more routinely used when PACS becomes available to link all our imaging equipments.