Fig 3 - uploaded by Mathieu Salzmann
Content may be subject to copyright.
![Kicking action from the HDM05 motion capture sequences database [14].](profile/Mathieu-Salzmann/publication/263699451/figure/fig2/AS:296041556529153@1447593215064/Kicking-action-from-the-HDM05-motion-capture-sequences-database-14.png)
Source publication
![Fig. 4: Samples from the FERET dataset [17].](profile/Mathieu-Salzmann/publication/263699451/figure/fig3/AS:296041556529154@1447593215118/Samples-from-the-FERET-dataset-17_Q320.jpg)
Representing images and videos with Symmetric Positive Definite (SPD)
matrices and considering the Riemannian geometry of the resulting space has
proven beneficial for many recognition tasks. Unfortunately, computation on the
Riemannian manifold of SPD matrices --especially of high-dimensional ones--
comes at a high cost that limits the applicabili...
Context in source publication
Context 1
... from motion capture sequences using the HDM05 database [14]. This database contains the following 14 actions: 'clap above head', 'deposit floor', 'elbow to knee', 'grab high', 'hop both legs', 'jog', 'kick forward', 'lie down floor', 'rotate both arms backward', 'sit down chair', 'sneak', 'squat', 'stand up lie' and 'throw basketball' (see Fig. 3 for an example). The dataset provides the 3D locations of 31 joints over time acquired at the speed of 120 frames per second. We describe an action of a K joints skeleton observed over m frames by its joint covariance descriptor [7], which is an SPD matrix of size 3K × 3K. This matrix is computed as in Eq. 12 by taking o i as the ...
Similar publications
This paper proposes a generalized framework with joint normalization which learns lower-dimensional subspaces with maximum discriminative power by making use of the Riemannian geometry. In particular, we model the similarity/dissimilarity between subspaces using various metrics defined on Grassmannian and formulate dimen-sionality reduction as a no...
Citations
... In our evaluations, we have conducted a thorough evaluation of the proposed DeepHolo-Brain model, focusing on two key classification challenges in neuroscience: brain task recognition and disease diagnosis. Since the geometric patterns in FC matrix have been widely investigated in the neuroscience field (You & Park, 2021), our comparative analysis includes a range of state-ofthe-art SPD matrix learning methods: Covariance Discriminative Learning (CDL) (Wang et al., 2012), Log-Euclidean Metric Learning (LEML) (Huang et al., 2015), SPD Manifold Learning (SPDML) (Harandi et al., 2014), Affine-Invariant Metric (AIM) (Pennec et al., 2006), Riemannian Sparse Representation (RSR) (Harandi et al., 2012), DeepO2P network (Ionescu et al., 2015), and SPDNet (Huang & Van Gool, 2017). For each method, we utilize the source codes and meticulously fine-tuned parameters according to the specifications provided by their respective authors to ensure the integrity of our evaluation. ...
The human brain is a complex inter-wired system that emerges spontaneous functional fluctuations. In spite of tremendous success in the experimental neuroscience field, a system-level understanding of how brain anatomy supports various neural activities remains elusive. Capitalizing on the unprecedented amount of neuroimaging data, we present a physics-informed deep model to uncover the coupling mechanism between brain structure and function through the lens of data geometry that is rooted in the widespread wiring topology of connections between distant brain regions. Since deciphering the puzzle of self-organized patterns in functional fluctuations is the gateway to understanding the emergence of cognition and behavior, we devise a geometric deep model to uncover manifold mapping functions that characterize the intrinsic feature representations of evolving functional fluctuations on the Riemannian manifold. In lieu of learning unconstrained mapping functions, we introduce a set of graph-harmonic scattering transforms to impose the brain-wide geometry on top of manifold mapping functions, which allows us to cast the manifold-based deep learning into a reminiscent of MLP-Mixer architecture (in computer vision) for Riemannian manifold. As a proof-of-concept approach, we explore a neural-manifold perspective to understand the relationship between (static) brain structure and (dynamic) function, challenging the prevailing notion in cognitive neuroscience by proposing that neural activities are essentially excited by brain-wide oscillation waves living on the geometry of human connectomes, instead of being confined to focal areas.
... However, using the original CMs as features for classification is challenging because they reside on the Riemannian manifold RM, not in Euclidean space. The features extracted from the Riemannian tangent space have better discriminative properties and are easier to handle than the original Riemannian space features [49]. This is primarily due to their linear properties and enhanced discriminability. ...
Emotion recognition is crucial in understanding human affective states with various applications. Electroencephalography (EEG)—a non-invasive neuroimaging technique that captures brain activity—has gained attention in emotion recognition. However, existing EEG-based emotion recognition systems are limited to specific sensory modalities, hindering their applicability. Our study innovates EEG emotion recognition, offering a comprehensive framework for overcoming sensory-focused limits and cross-sensory challenges. We collected cross-sensory emotion EEG data using multimodal emotion simulations (three sensory modalities: audio/visual/audio-visual with two emotion states: pleasure or unpleasure). The proposed framework—filter bank adversarial domain adaptation Riemann method (FBADR)—leverages filter bank techniques and Riemannian tangent space methods for feature extraction from cross-sensory EEG data. Compared with Riemannian methods, filter bank and adversarial domain adaptation could improve average accuracy by 13.68% and 8.36%, respectively. Comparative analysis of classification results proved that the proposed FBADR framework achieved a state-of-the-art cross-sensory emotion recognition performance and reached an average accuracy of 89.01% ± 5.06%. Moreover, the robustness of the proposed methods could ensure high cross-sensory recognition performance under a signal-to-noise ratio (SNR) ≥ 1 dB. Overall, our study contributes to the EEG-based emotion recognition field by providing a comprehensive framework that overcomes limitations of sensory-oriented approaches and successfully tackles the difficulties of cross-sensory situations.
... In many recent studies, representing the raw data using SPD matrices and taking into account their specific Riemannian geometry have shown promising results, e.g. in computer vision [7,83], for domain adaptation [85], on medical data [3] and in recognition tasks [39]. For example, Barachant et al. [3] proposed a support-vector-machine (SVM) classifier that takes into account the Riemannian geometry of the features, which are SPD covariance matrices, representing Electroencephalogram (EEG) recordings. ...
... There exists a few dimension reduction methods targeting for the Riemannian space of SPD matrices. Most of them attempt to learn a mapping that projects the high-dimensional SPD matrices into a lower-dimensional space [9,10], following the principle of retaining the distance structure in the original space as much as possible [11]. In contrast, data points that are proximal in the high-dimensional space stay close in the projected space, while data points that are distant in the high-dimensional space remain far away from each other in the low-dimensional space. ...
... Surprisingly, in downstream processing and analysing of manifold-valued data -e.g., dimension reduction for visualisation [33] or for retrieving underlying structure from (principal) sub-manifolds [27] -the symmetric structure is hardly ever used. Typically, works seem to choose between methods that are either global, but manifold-specific [34] or localized to a tangent space, but general [28]. Focusing on the latter class of approaches, subsequently quantifying the localization error is hardly ever addressed. ...
... will be close to the curvature corrected solution of (34). In particular, we will see that a minimizer of (34) converges to the naive solution under vanishing curvature. In other words, the naive solution can be useful as a starting point for a (possibly iterative) curvature correction scheme based on a nicer minimization problem (34) than the one based on minimizing the global approximation error (33). ...
... The above observation motivates one to consider the relation between minimizers of the curvature corrected approximation problem (34) and the naive problem (35). A natural question to ask is whether the minimizers of (35) are stable under curvature effects. ...
When generalizing schemes for real-valued data approximation or decomposition to data living in Riemannian manifolds, tangent space-based schemes are very attractive for the simple reason that these spaces are linear. An open challenge is to do this in such a way that the generalized scheme is applicable to general Riemannian manifolds, is global-geometry aware and is computationally feasible. Existing schemes have been unable to account for all three of these key factors at the same time. In this work, we take a systematic approach to developing a framework that is able to account for all three factors. First, we will restrict ourselves to the -- still general -- class of symmetric Riemannian manifolds and show how curvature affects general manifold-valued tensor approximation schemes. Next, we show how the latter observations can be used in a general strategy for developing approximation schemes that are also global-geometry aware. Finally, having general applicability and global-geometry awareness taken into account we restrict ourselves once more in a case study on low-rank approximation. Here we show how computational feasibility can be achieved and propose the curvature-corrected truncated higher-order singular value decomposition (CC-tHOSVD), whose performance is subsequently tested in numerical experiments with both synthetic and real data living in symmetric Riemannian manifolds with both positive and negative curvature.
... Regarding the SPD manifold, we derived proof for a limited set of Riemannian metrics to be EPMs. However, many other metrics than the ones considered in Lemma (3.4) could also be EPMs [14,48]. Not all Riemannian metrics can be EPMs, but we can combine these results with classical results in Riemannian Geometry, such as the Nash Embedding Theorem [49], to study the general properties of an arbitrary Riemannian metric. ...
The paper introduces a novel non-parametric Riemannian regression method using Isometric Riemannian Manifolds (IRMs). The proposed technique, Intrinsic Local Polynomial Regression on IRMs (ILPR-IRMs), enables global data mapping between Riemannian manifolds while preserving underlying geometries. The ILPR method is generalized to handle multivariate covariates on any Riemannian manifold and isometry. Specifically, for manifolds equipped with Euclidean Pullback Metrics (EPMs), a closed analytical formula is derived for the multivariate ILPR (ILPR-EPM). Asymptotic statistical properties of the ILPR-EPM for the multivariate local linear case are established, including a formula for the asymptotic bias, establishing estimator consistency. The paper showcases possible applications of the method by focusing on a group of Riemannian metrics on the Symmetric Positive Definite (SPD) manifold, which arises in machine learning and neuroscience. It is demonstrated that several metrics on the SPD manifold are EPMs, resulting in a closed analytical expression for the multivariate ILPR estimator on the SPD manifold. The paper evaluates the ILPR estimator's performance under two specific EPMs, Log-Cholesky and Log-Euclidean, on simulated data on the SPD manifold and compares it with extrinsic LPR using the Affine-Invariant when scaling the manifold and covariate dimension. The results show that the ILPR using the Log-Cholesky metric is computationally faster and provides a better trade-off between error and time than other metrics. Finally, the Log-Cholesky metric on the SPD manifold is employed to implement an efficient and intrinsic version of Rie-SNE for visualizing high-dimensional SPD data. The code for implementing ILPR-EPMs and other relevant calculations is available on the GitHub page.
... The inherent dimension of the structure is m = 10, and the actual dimension of the structure is n = 30. A brief generation scheme used in [47] is as follows: The high-dimensional data consists of two parts: inherent feature vector s fea i ∈ R m(m+1)/2 and auxiliary vector s ...
In brain–computer interface (BCI)-based motor imagery, the symmetric positive definite (SPD) covariance matrices of electroencephalogram (EEG) signals with discriminative information features lie on a Riemannian manifold, which is currently attracting increasing attention. Under a Riemannian manifold perspective, we propose a non-linear dimensionality reduction algorithm based on neural networks to construct a more discriminative low-dimensional SPD manifold. To this end, we design a novel non-linear shrinkage layer to modify the extreme eigenvalues of the SPD matrix properly, then combine the traditional bilinear mapping to non-linearly reduce the dimensionality of SPD matrices from manifold to manifold. Further, we build the SPD manifold network on a Siamese architecture which can learn the similarity metric from the data. Subsequently, the effective signal classification method named minimum distance to Riemannian mean (MDRM) can be implemented directly on the low-dimensional manifold. Finally, a regularization layer is proposed to perform subject-to-subject transfer by exploiting the geometric relationships of multi-subject. Numerical experiments for synthetic data and EEG signal datasets indicate the effectiveness of the proposed manifold network.
... SPD manifold achieves great success in computer vision due to its powerful statistical representations for images and videos. For example, SPD matrices are used to construct region covariance matrices for pedestrian detection [31], joint covariance descriptors for action recognition [32], and image set covariance matrices for face recognition [33]. ...
... Table 5 shows that SPDNet [50] and GrNet [51] can achieve better classification results than state-of-the-art methods on AFEW dataset [99]. The following methods for comparison are shallow learning methods applying manifold structure: Expressionlets on Spatio-Temporal Manifold (STM-ExpLet) [107], Riemannian Sparse Representation combining with Manifold Learning on the manifold of SPD matrices (RSR-SPDML) [32], Discriminative Canonical Correlations (DCC) [109], Graßmann Discriminant Analysis (GDA) [56], Grassmannian Graph-Embedding Discriminant Analysis (GGDA) [118], and Projection Metric Learning (PML) [110]. Deep Second-order Pooling (DeepO2P) [108] is a traditional CNN model using the standard optimization method. ...
... However, considering unexplored deep learning methods such as reinforcement learning, together with unused manifold structures such as centered matrix manifold, it is still a huge challenge to push the boundaries of geometric optimization in deep learning. Sphere, ellipsoid, SPD (n), Landmarks, GL (n), SO (n), SE (n) Inner product, exponential and logarithmic maps, parallel transport, Christoffel symbols, Riemann, Ricci and scalar curvature, geodesics, Fréchet mean MNIST [98] uRNN [14] 97.6% full-capacity uRNN [69] 96.9% expRNN [13] 98.7% soRNN [72] 97.3% ORNN [73] 97.2% GORU [74] 98.9% RSR-SPDML [32] 48.01% DCC [109] 41.74% GDA [56] 46.25% GGDA [118] 46.87% PML [110] 47.25% SPDNet [50] 61.45% GrNet [51] 59.23% 78.82% 68.24% DeepO2P [108] 68.76% 60.14% DCC [109] 75.83% 67.04% GDA [56] 71.38% 67.49% GGDA [118] 66.71% 68.41% PML [110] 73.45% 68.32% SPDNet [50] 80.12% 72.83% GrNet [51] 80.52% 72.76% Lazebnik et al. [120] 81.2% Dixit et al. [121] 82.3% Kwitt et al. [122] 85.4% Goh et al. [123] 85.4% SRMR [60] 86.9% ...
Although Deep Learning (DL) has achieved success in complex Artificial Intelligence (AI) tasks, it suffers from various notorious problems (e.g., feature redundancy, and vanishing or exploding gradients), since updating parameters in Euclidean space cannot fully exploit the geometric structure of the solution space. As a promising alternative solution, Riemannian-based DL uses geometric optimization to update parameters on Riemannian manifolds and can leverage the underlying geometric information. Accordingly, this article presents a comprehensive survey of applying geometric optimization in DL. At first, this article introduces the basic procedure of the geometric optimization, including various geometric optimizers and some concepts of Riemannian manifold. Subsequently, this article investigates the application of geometric optimization in different DL networks in various AI tasks, e.g., convolution neural network, recurrent neural network, transfer learning, and optimal transport. Additionally, typical public toolboxes that implement optimization on manifold are also discussed. Finally, this article makes a performance comparison between different deep geometric optimization methods under image recognition scenarios.
... In image processing, region information of an image can be effectively captured through several SPD covariance descriptors (Tuzel et al., 2006). For the application of image set/video classification, each set of images/frames can be represented by its covariance matrix, which has shown great promise in modelling the intra-set variations (Huang et al., 2015;Harandi et al., 2014). In addition, fields of SPD matrices are also important in computer graphics for anisotropic diffusion (Weickert, 1998), remeshing (Alliez et al., 2003 and texture synthesis (Galerne et al., 2010), just to name a few. ...
... Then, while (A) is a non-negative scalar, the "mass" M(A) ∈ d + , where d + denotes the set of d × d positive semi-definite matrices. SPD matrix-valued measures have been employed in applications such as diffusion tensor imaging (Le Bihan et al., 2001), image set classification (Huang et al., 2015;Harandi et al., 2014), anisotropic diffusion (Weickert, 1998), and brain imaging (Assaf and Pasternak, 2008), to name a few. ...
... Here, we perform the experiments of domain adaptation on more challenging tasks, including video based face recognition with YouTube Celebrities (YTC) dataset (Kim et al., 2008) and texture classification via Dynamic Texture (DynTex) (Ghanem and Ahuja, 2010) dataset, where covariance representation learning has shown great promise Huang et al. (2015); Harandi et al. (2014). ...
In this work, we study the optimal transport (OT) problem between symmetric positive definite (SPD) matrix-valued measures. We formulate the above as a generalized optimal transport problem where the cost, the marginals, and the coupling are represented as block matrices and each component block is a SPD matrix. The summation of row blocks and column blocks in the coupling matrix are constrained by the given block-SPD marginals. We endow the set of such block-coupling matrices with a novel Riemannian manifold structure. This allows to exploit the versatile Riemannian optimization framework to solve generic SPD matrix-valued OT problems. We illustrate the usefulness of the proposed approach in several applications.
... However, this requirement is hard to be satisfied in some machine learning tasks e.g. RKHS transporting and dimension reduction [18], [19], where rank deficient matrices are frequently encountered. To account for this, we establish the following theorem: ...
The Matrix-based Renyi's entropy enables us to directly measure information quantities from given data without the costly probability density estimation of underlying distributions, thus has been widely adopted in numerous statistical learning and inference tasks. However, exactly calculating this new information quantity requires access to the eigenspectrum of a semi-positive definite (SPD) matrix $A$ which grows linearly with the number of samples $n$, resulting in a $O(n^3)$ time complexity that is prohibitive for large-scale applications. To address this issue, this paper takes advantage of stochastic trace approximations for matrix-based Renyi's entropy with arbitrary $\alpha \in R^+$ orders, lowering the complexity by converting the entropy approximation to a matrix-vector multiplication problem. Specifically, we develop random approximations for integer order $\alpha$ cases and polynomial series approximations (Taylor and Chebyshev) for non-integer $\alpha$ cases, leading to a $O(n^2sm)$ overall time complexity, where $s,m \ll n$ denote the number of vector queries and the polynomial order respectively. We theoretically establish statistical guarantees for all approximation algorithms and give explicit order of s and m with respect to the approximation error $\varepsilon$, showing optimal convergence rate for both parameters up to a logarithmic factor. Large-scale simulations and real-world applications validate the effectiveness of the developed approximations, demonstrating remarkable speedup with negligible loss in accuracy.
