No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Characterizing the variability of footstep-induced structural vibrations for open-world person identification
Abstract
Person identification is important in providing personalized services in smart buildings. Many existing studies focus on closed-world person identification, which only identifies a fixed group of people who have training data; however, they assume everyone has pre-collected data, which is not practical in real-world scenarios when newcomers are present. To overcome this drawback, open-world person identification recognizes both newcomers and registered people, which opens up new opportunities for smart building applications that involve newcomers, such as smart visitor management, customized retail, personalized health monitoring, and public emergency assistance. To achieve this, structural vibration sensing has various advantages when compared with the existing sensing modalities (e.g., cameras, wearables, and pressure sensors) because it only needs sparsely deployed sensors mounted on the floor, does not require people to carry devices, and is perceived as more privacy-friendly. However, one fundamental challenge in analyzing footstep-induced structural vibration data is its high variability due to the structural heterogeneity and the footstep variations. Therefore, it is difficult to distinguish different people given this high variability within each person, and it is more challenging to recognize a new person as that data is unobserved before. In this paper, we characterize the variability in footstep-induced structural vibration to develop an open-world person identification framework. Specifically, we address three variability challenges in developing our method. First, the high variability within each person comes from multiple sources that are entangled in the vibration signals, and thus is difficult to be decomposed and reduced. Secondly, the distribution of features extracted from the vibration signals is irregularly shaped, and therefore is difficult to model. Moreover, the identity of the next person is correlated with the previous observations, which makes the identification process more complicated. To overcome these challenges, we first characterize multiple variability sources and design a transformation function that results in signal features that are less variable within one person and more separable between different people. We then develop a modified Chinese Restaurant Process (mCRP) for nonparametric Bayesian modeling to capture the irregularly shaped feature patterns both from local and global perspectives. Finally, we design an adaptive hyperparameter that represents the prior probability of newcomers at each observation, which keeps updating depending on the time, location, and previous predictions. We evaluate our approach through walking experiments with 20 people across 2 different structures. With only 1 pre-recorded person at each structure, our method achieves up to 92.3% average accuracy with randomly appearing newcomers.
... The main benefits of using floor vibration to infer human behaviors are that it is cost-efficient, allows continuous, fine-grained crowd behavior monitoring, and is perceived as more privacy-friendly than cameras or audio recordings. This sensing approach has been explored in many existing applications, such as occupant detection [26,31], identification [10,30], activity recognition [3,29], localization [27], and health monitoring [11,12,20]. ...
... The potential of using structural vibrations to infer behaviors of humans or animals has been explored in many previous studies. Our prior work has shown promise in using the footstep-induced floor vibrations for occupancy detection [26,31], identification [10,30], gait health monitoring [2,11,12,20]. In addition to footsteps, vibrations induced by human activities can also be used for the prediction and characterization of activity types and patterns [3,7,29]. ...
... The floor vibration induced by human gait is not only influenced by gait patterns, but also by the floor properties, shoe types, and the participants' body configurations 43 . For example, existing studies have found that the difference in floor properties results in shifts in vibration frequency ranges due to the changes in stiffness and mass of the floor material 41,44 . ...
Muscular dystrophies (MD) are a group of genetic neuromuscular disorders that cause progressive weakness and loss of muscles over time, influencing 1 in 3500–5000 children worldwide. New and exciting treatment options have led to a critical need for a clinical post-marketing surveillance tool to confirm the efficacy and safety of these treatments after individuals receive them in a commercial setting. For MDs, functional gait assessment is a common approach to evaluate the efficacy of the treatments because muscle weakness is reflected in individuals’ walking patterns. However, there is little incentive for the family to continue to travel for such assessments due to the lack of access to specialty centers. While various existing sensing devices, such as cameras, force plates, and wearables can assess gait at home, they are limited by privacy concerns, area of coverage, and discomfort in carrying devices, which is not practical for long-term, continuous monitoring in daily settings. In this study, we introduce a novel functional gait assessment system using ambient floor vibrations, which is non-invasive and scalable, requiring only low-cost and sparsely deployed geophone sensors attached to the floor surface, suitable for in-home usage. Our system captures floor vibrations generated by footsteps from patients while they walk around and analyzes such vibrations to extract essential gait health information. To enhance interpretability and reliability under various sensing scenarios, we translate the signal patterns of floor vibration to pathological gait patterns related to MD, and develop a hierarchical learning algorithm that aggregates insights from individual footsteps to estimate a person’s overall gait performance. When evaluated through real-world experiments with 36 subjects (including 15 patients with MD), our floor vibration sensing system achieves a 94.8% accuracy in predicting functional gait stages for patients with MD. Our approach enables accurate, accessible, and scalable functional gait assessment, bringing MD progressive tracking into real life.
... Henceforth, the terms vibro-localization and vibro-measurements will refer to such localization techniques and the measurements used in these techniques, respectively. Vibro-localization techniques facilitate a myriad of applications ranging from smart home monitoring and event classification [1][2][3][4][5] to human gait assessment [6][7][8][9][10][11][12] and occupant identification and tracking [13][14][15][16][17][18][19][20]. ...
This paper presents an occupant localization technique that determines the location of individuals in indoor environments by analyzing the structural vibrations of the floor caused by their footsteps. Structural vibration waves are difficult to measure as they are influenced by various factors, including the complex nature of wave propagation in heterogeneous and dispersive media (such as the floor) as well as the inherent noise characteristics of sensors observing the vibration wavefronts. The proposed vibration-based occupant localization technique minimizes the errors that occur during the signal acquisition time. In this process, the likelihood function of each sensor—representing where the occupant likely resides in the environment—is fused to obtain a consensual localization result in a collective manner. In this work, it becomes evident that the above sources of uncertainties can render certain sensors deceptive, commonly referred to as “Byzantines.” Because the ratio of Byzantines among the set sensors defines the success of the collective localization results, this paper introduces a Byzantine sensor elimination (BSE) algorithm to prevent the unreliable information of Byzantine sensors from affecting the location estimations. This algorithm identifies and eliminates sensors that generate erroneous estimates, preventing the influence of these sensors on the overall consensus. To validate and benchmark the proposed technique, a set of previously conducted controlled experiments was employed. The empirical results demonstrate the proposed technique’s significant improvement (3~0%) over the baseline approach in terms of both accuracy and precision.
Precision Swine Farming has the potential to directly benefit swine health and industry profit by automatically monitoring the growth and health of pigs. We introduce the first system to use structural vibration to track animals, and the first system for automated characterization of piglet group activities, including nursing, sleeping, and active times. PigSense uses physical knowledge of the structural vibration characteristics caused by pig-activity-induced load changes to recognize different behaviors of the sow and piglets. In order for our system to survive the harsh environment of the farrowing pen for three months, we designed simple, durable sensors for physical fault tolerance, then installed many of them, pooling their data to achieve algorithmic fault tolerance even when some do stop working. The key focus of this work was to create a robust system that can withstand challenging environments, has limited installation and maintenance requirements, and uses domain knowledge to precisely detect a variety of swine activities in noisy conditions while remaining flexible enough to adapt to future activities and applications. We provided an extensive analysis and evaluation of all-round swine activities and scenarios from our 1-year field deployment across two pig farms in Thailand and the USA. To help assess the risk of crushing, farrowing sicknesses, and poor maternal behaviors, PigSense achieves an average of 97.8% and 94% for sow posture and motion monitoring, respectively, and an average of 96% and 71% for ingestion and excretion detection. To help farmers monitor piglet feeding, starvation, and illness, PigSense achieves an average of 87.7%, 89.4%, and 81.9% in predicting different levels of nursing, sleeping, and being active, respectively. In addition, we show that our monitoring of signal energy changes allows the prediction of farrowing in advance, as well as status tracking during the farrowing process and on the occasion of farrowing issues. Furthermore, PigSense also predicts the daily pattern and weight gain in the lactation cycle with 89% accuracy, a metric that can be used to monitor the piglets’ growth progress over the lactation cycle.
In-home gait analysis is important for providing early diagnosis and adaptive treatments for individuals with gait disorders. Existing systems include wearables and pressure mats, but they have limited scalability due to dense deployment and device carrying/charging requirements. Recently, vision-based systems have been developed to enable scalable, accurate in-home gait analysis, but it faces privacy concerns due to the exposure of people's appearances and daily activities. To overcome these limitations, our prior work developed footstep-induced structural vibration sensing for in-home gait monitoring, which is device-free, wide-ranged, and perceived as more privacy-friendly. Although it has succeeded in temporal parameter estimation, it shows limited performance for spatial gait parameter estimation due to the low accuracy in footstep localiza-tion. In particular, the localization error mainly comes from the estimation error of the wave arrival time at the vibration sensors and its error propagation to wave velocity estimations. To this end, we present GaitVibe+, a vibration-based footstep localization method fused with temporarily installed cameras for in-home gait analysis. Our method has two stages: fusion and operating stages. In the fusion stage, both cameras and vibration sensors are installed to record only a few trials of the subject's footstep data, through which we characterize the uncertainty in wave arrival time and model the wave velocity profiles for the given structure. In the operating stage, we remove the camera to preserve privacy at home. The footstep localization is conducted by estimating the time difference of arrival (TDoA) over multiple vibration sensors, whose accuracy is improved through the reduced uncertainty and velocity modeling during the fusion stage. We evaluate GaitVibe+ through a real-world experiment with 50 walking trials. With only 3 trials of multi-modal fusion, our approach has an average localization error of 0.22 meters, which reduces the spatial gait parameter error by 4.1x (from 111.4% to 27.1%) compared to the existing work.
Human-induced structural vibration provides valuable information about humans who interact with the structure, including people’s identity, characteristics, activities and health status. Among them, person identification is crucial because it is the premise for any personalized services. Our prior work using footstep-induced structural vibration has been promising in identifying a fixed group of people, but it is restricted to registered occupants, leading to errors whenever a stranger appears. Therefore, we introduce a stranger detection and occupant identification system using footstep-induced structural vibrations. There are two main challenges in developing this system: 1) the strangers’ walking patterns are unknown before being observed, and 2) the probability of the presence of a stranger varies at different times and locations. To overcome the first challenge, we model the occupants’ and strangers’ footstep-induced floor vibration data as a mixture of Gaussian distributions with varying number of components, representing the changing number of registered people. To address the second challenge, we reformulated the prior estimation in the mixture model by introducing an interpretable parameter that represents the expected probability of observing a stranger. Through an experiment conducted on a wood-framed structural platform, our method achieves an average accuracy of 89.2% in person identification among 10 people.KeywordsPerson identificationStranger detectionStructural vibrationsDirichlet process
In sensed buildings, information related to occupant movement helps optimize functions such as security, energy management, and caregiving. Due to privacy needs, non-intrusive sensing approaches for tracking occupants inside buildings, such as vibration sensors, are often preferred over intrusive strategies that involve use of cameras and wearable devices. Current sensor-based occupant-localization approaches are data-driven techniques that do not account for structural behavior and limited to slabs on grade. Varying-rigidity floors and inherent variability in walking gaits lead to ambiguous interpretations of floor vibrations when performing model-free occupant localization. In this paper, an extensive analysis of vibrations induced by a range of occupants is described. Firstly, the need for a structural-behavior model for occupant localization is assessed using two full-scale case studies. Structural behavior is found to significantly influence floor vibrations induced by footstep impacts. Since a simple relationship between distances from footstep-impact to sensor locations cannot be assured, the use of physics-based models is necessary for accurate occupant localization. Secondly, measured data are interpreted using physics-based models and information related to uncertainties from multiple sources. There are two types of uncertainties: modelling uncertainties and measurement uncertainties, including variability in walking gaits. Error-domain model falsification (EDMF) and residual minimization (RM) are model-based approaches for data interpretation. Unlike RM, EDMF explicitly accounts for the presence of systematic errors in parameters and overall model bias. In this paper, model-based occupant localization is carried out using EDMF and RM on a full-scale case study. By explicitly accounting for the presence of uncertainties and the influence of structural behavior, EDMF, unlike RM, accurately reveals possible occupant locations on floor slabs.
In this paper, we characterize the effects of obstructions on footstep-induced floor vibrations to enable obstruction-invariant indoor occupant localization. Occupant localization is important in smart building applications such as smart healthcare and energy management. Maintenance and installment requirements limit the application of current sensing approaches (e.g., mobile-based, RF-based, and pressure-based sensing) in real-life applications. To overcome these limitations, prior work has utilized footstep-induced structural vibrations for occupant localization. The main intuition behind these approaches is that the footstep-induced floor vibration waves take different amounts of time to arrive at different sensors. These Time-Differences-of-Arrival (TDoA) can then be leveraged to locate the footstep by assuming similar velocities between the footstep and various sensor locations. This assumption makes these approaches suitable for open areas; however, real buildings have various types of obstructions (e.g., walls, furniture, etc.) which affect wave propagation velocities and hence significantly reduce localization accuracy. Therefore, the prior work requires unobstructed paths between footsteps and sensors for accurate occupant localization, which increases the sensing density requirement and thus, instrumentation and maintenance costs. We have observed that the obstruction mass is one of the key factors in affecting the wave propagation velocity and reducing the localization accuracy. Therefore, to overcome the obstruction challenge, we localize footsteps by considering different velocities between the footsteps and sensors depending on the existence and mass of obstruction on the wave path. Specifically, we (1) detect and estimate the mass of the obstruction by characterizing the wave attenuation rate, (2) use this estimated mass to find the propagation velocities for localization by modeling the velocity-mass relationship through the lamb wave characteristics, and (3) introduce a non-isotropic multilateration approach which robustly leverages these propagation velocities to locate the footsteps (and the occupants). In field experiments, we achieved average localization error of 0.61 meters, which is (1) the same as the average localization error when there is no obstruction and (2) 1.6X improvement compared to the baseline approach.
In this work, we propose a novel unconstrained biometric authentication system that utilizes footstep generated seismic waves. It has the following advantages over existing systems: i) it doesn't require any special orientation or positioning of the individual for authentication, and ii) seismic data do not interfere with the individual's privacy. We curate an indigenous dataset containing ∼ 78000 footstep events collected from 8 individuals in three types of surfaces (concrete tile floor, carpet floor, and wooden floor) to validate the robustness of the proposed system. It is also capable of detecting unauthorized/unregistered individuals (imposters). To mimic realistic scenarios for imposter detection, the models are trained only with the footstep data of registered users (footsteps of unregistered users are completely unknown/unseen to the trained models), thus achieving imposter detection for footstep based person identification. We achieved an accuracy of 90% (in concrete tile floor and wooden floor) and 94% (in carpet floor) for person identification by using features from as low as two consecutive footsteps. For imposter detection, we achieved an accuracy of 76%-80% (in concrete tile floor and wooden floor) and 87% (in carpet floor) by utilizing features from 10 consecutive footsteps.
We introduce a footstep-induced floor vibration sensing system that enables us to quantify the gait pattern of individuals with Muscular Dystrophy (MD) in non-clinical settings. MD is a neuromuscular disorder causing progressive loss of muscle, which leads to symptoms in gait patterns such as toe-walking, frequent falls, balance difficulty, etc. Existing systems that are used for progressive tracking include pressure mats, wearable devices, or direct observation by healthcare professionals. However, they are limited by operational requirements including dense deployment, users' device carrying, special training, etc. To overcome these limitations, we introduce a new approach that senses floor vibrations induced by human footsteps. Gait symptoms in these footsteps are reflected by the vibration signals, which enables monitoring of gait health for individuals with MD. Our approach is non-intrusive, unrestricted by line-of-sight, and thus suitable for in-home deployment. To develop our approach, we characterize the gait pattern of individuals with MD using vibration signals, and infer the health state of the patients based on both symptom-based and signal-based features. However, there are two main challenges: 1) different aspects of human gaits are mixed up in footstep-induced floor vibrations; and 2) structural heterogeneity distorts vibration propagation and attenuation through the floor medium. To overcome the first challenge, we characterize the symptom-based gait features of the footstep-induced floor vibration specific to MD. To minimize the performance inconsistency across different sensing locations in the building, we reduce the structural effects by removing the free-vibration phase due to structural damping. With these two challenges addressed, we evaluate our system performance by conducting a real-world experiment with six patients with MD and seven healthy participants. Our approach achieved 96% accuracy in predicting whether the footstep was from a patient with MD.
Sensor-based occupant tracking has the potential to enhance knowledge of the utilization of buildings. Occupancy-tracking strategies using footstep-induced floor vibrations may be beneficial for thermal-load prediction, security enhancement, and care-giving without undermining privacy. Current floor-vibration-based occupant-tracking methodologies are based on data-driven techniques that do not include a physics-based model of the structural behavior of the floor slab. These techniques suffer from ambiguous interpretations when signals are affected by complex configurations of structural and non-structural elements such as beams and walls. Using a physics-based model for data-interpretation enables deployment of sparse number of sensors in contexts of non-uniform structural configurations. In this paper, an application of physics-based data interpretation using error-domain model falsification (EDMF) is presented to track an occupant within an office environment through footstep-induced floor vibrations. EDMF is a population-based approach that incorporates various sources of uncertainty, including bias, arising from measurements and modeling. EDMF involves the rejection of simulated model responses that contradict footstep-induced floor vibration measurements. Thus, EDMF provides a set of candidate locations from an initial population of possible occupant locations. A sequential analysis that accommodates information from previous footsteps is then used to enhance candidate locations and identify trajectories among candidates. In this way, incorporating structural behavior in interpreting vibration measurements induced by occupant footsteps has the potential to identify accurately the trajectory of an occupant in buildings with complex configurations, thereby providing tracking information without undermining privacy.
Human footsteps can provide a unique behavioural pattern for robust biometric systems. We propose spatio-temporal footstep representations from floor-only sensor data in advanced computational models for automatic biometric verification. Our models deliver an artificial intelligence capable of effectively differentiating the fine-grained variability of footsteps between legitimate users (clients) and impostor users of the biometric system. The methodology is validated in the largest to date footstep database, containing nearly 20,000 footstep signals from more than 120 users. The database is organized by considering a large cohort of impostors and a small set of clients to verify the reliability of biometric systems. We provide experimental results in 3 critical data-driven security scenarios, according to the amount of footstep data made available for model training: at airports security checkpoints (smallest training set), workspace environments (medium training set) and home environments (largest training set). We report state-of-the-art footstep recognition rates with an optimal equal false acceptance and false rejection rate of 0.7% (equal error rate), an improvement ratio of 371% from previous state-of-the-art. We perform a feature analysis of deep residual neural networks showing effective clustering of client's footstep data and provide insights of the feature learning process.
We present FootprintID, an indoor pedestrian identification system that utilizes footstep-induced structural vibration to infer pedestrian identities for enabling various smart building applications. Previous studies have explored other sensing methods, including vision-, RF-, mobile-, and acoustic-based methods. They often require specific sensing conditions, including line-of-sight, high sensor density, and carrying wearable devices. Vibration-based methods, on the other hand, provide easy-to-install sparse sensing and utilize gait to distinguish different individuals. However, the challenge for these methods is that the signals are sensitive to the gait variations caused by different walking speeds and the floor variations caused by structural heterogeneity.
We present FootprintID, a vibration-based approach that achieves robust pedestrian identification. The system uses vibration sensors to detect footstep-induced vibrations. It then selects vibration signals and classifiers to accommodate sensing variations, taking step location and frequency into account. We utilize the physical insight on how individual step signal changes with walking speeds and introduce an iterative transductive learning algorithm (ITSVM) to achieve robust classification with limited labeled training data. When trained only on the average walking speed and tested on different walking speeds, FootprintID achieves up to 96% accuracy and a 3X improvement in extreme speeds compared to the Support Vector Machine. Furthermore, it achieves up to 90% accuracy (1.5X improvement) in uncontrolled experiments.
Sensor-based occupant detection has the potential to make an important contribution to the development of structures of the future. Applications that may benefit from robust occupant detection include patient detection in hospitals, senior citizen housing facilities, personnel localization in emergencies as well as user behavior studies. In this contribution, an occupant detection and localization methodology based on recorded vibration time-series is outlined. The movement of an occupant on a floor generates vibrations that can be recorded by accelerometers. However, measured vibrations contain measurement noise and are contaminated by ambient sources of vibration such as machinery and nearby traffic. This contribution relies on using filtered vibration time-series to detect events of moving occupants and subsequently perform model-based localization of occupants using error-domain model falsification. The error-domain model-falsification methodology utilizes multiple models to deal with ambiguity related to the inverse problem of occupant localization. By explicitly incorporating uncertainty from various sources using engineering heuristics, error-domain model falsification provides a set of candidate locations based on measurements obtained through a coarse sensor configuration. The results from this methodology provide in a binary manner the presence or absence of an occupant and subsequently candidate locations of the occupant on the floor of a 200 m² hall that is equipped with only four accelerometers.
Touch surfaces are intuitive interfaces for computing devices. Most of the traditional touch interfaces (vision, IR, capacitive, etc.) have mounting requirements, resulting in specialized touch surfaces limited by their size, cost, and mobility. More recent work has shown that vibration-based touch sensing techniques can localize taps/knocks, which provides a low-cost flexible alternative. These surfaces are envisioned as intuitive inputs for applications such as interactive meeting tables, smart kitchen appliance control, etc. However, due to dispersive and reflective properties of various vibrating mediums, it is difficult to localize taps accurately on ubiquitous surfaces. Furthermore, no work has been done on tracking continuous swipe interactions through vibration sensing.
In this paper, we present SurfaceVibe, a vibration-based interaction tracking system for multiple surface types. The system accounts for physics properties of different waves to allow two major interaction types: tap and swipe. For tap induced impulse-like surface waves, we design an algorithm that takes wave dispersion and reflection into account to achieve accurate localization on ubiquitous surfaces. For swipe induced body waves, SurfaceVibe segments signals into 'slip pulses' to localize, and then tracks the trajectory. We validate SurfaceVibe through experiments on different materials and varying surface/sensing area sizes in this paper. Our methods achieve up to 6X decrease in localization error for taps and 3X reduction in length estimation error for swipes compared to existing algorithms that do not take wave properties into account.
The spectral approach is employed for spectral analysis of a beam subjected to an arbitrary force traveling along the beam with constant speed. First, an expression for exact frequency response of a beam subjected to moving arbitrary force and general boundary conditions has been constructed. The obtained frequency domain response allows straightforwardly exhibiting response vibration components governed by different frequencies such as the natural, loading and driving ones and their interaction. This provides also alternative insight to the cancellation of response at natural frequency. The theoretical development is illustrated and validated by numerical examination on a simply supported beam under moving harmonic forces.
Person identification is an important but still challenging problem in video surveillance. This work designs a completely automatic appearance-based person identification system, which has the ability to achieve new person discovery and classification. The proposed system consists of three modules: background and silhouette separation; feature extraction and selection; and online person identification. The Self-Adaptive Kernel Machine (SAKM) algorithm is used to differentiate existing persons who can be classified from new persons who have to be learnt and added. A new video database with 22 persons is created in real-life environments. The experimental results show that the proposed system achieves satisfying recognition rates of over 90% on person classification with novelty identification.
The key challenge of face recognition is to develop effective feature
representations for reducing intra-personal variations while enlarging
inter-personal differences. In this paper, we show that it can be well solved
with deep learning and using both face identification and verification signals
as supervision. The Deep IDentification-verification features (DeepID2) are
learned with carefully designed deep convolutional networks. The face
identification task increases the inter-personal variations by drawing DeepID2
extracted from different identities apart, while the face verification task
reduces the intra-personal variations by pulling DeepID2 extracted from the
same identity together, both of which are essential to face recognition. The
learned DeepID2 features can be well generalized to new identities unseen in
the training data. On the challenging LFW dataset, 99.15% face verification
accuracy is achieved. Compared with the best deep learning result on LFW, the
error rate has been significantly reduced by 67%.
Advances in computation and the fast and cheap computational facilities now available to statisticians have had a significant impact upon statistical research, and especially the development of nonparametric data analysis procedures. In particular, theoretical and applied research on nonparametric density estimation has had a noticeable influence on related topics, such as nonparametric regression, nonparametric discrimination, and nonparametric pattern recognition. This article reviews recent developments in nonparametric density estimation and includes topics that have been omitted from review articles and books on the subject. The early density estimation methods, such as the histogram, kernel estimators, and orthogonal series estimators are still very popular, and recent research on them is described. Different types of restricted maximum likelihood density estimators, including order-restricted estimators, maximum penalized likelihood estimators, and sieve estimators, are discussed, where restrictions are imposed upon the class of densities or on the form of the likelihood function. Nonparametric density estimators that are data-adaptive and lead to locally smoothed estimators are also discussed; these include variable partition histograms, estimators based on statistically equivalent blocks, nearest-neighbor estimators, variable kernel estimators, and adaptive kernel estimators. For the multivariate case, extensions of methods of univariate density estimation are usually straightforward but can be computationally expensive. A method of multivariate density estimation that did not spring from a univariate generalization is described, namely, projection pursuit density estimation, in which both dimensionality reduction and density estimation can be pursued at the same time. Finally, some areas of related research are mentioned, such as nonparametric estimation of functionals of a density, robust parametric estimation, semiparametric models, and density estimation for censored and incomplete data, directional and spherical data, and density estimation for dependent sequences of observations.
This paper presents results obtained from ambient vibration measurements using a remote monitoring system (RMS) on a grandstand in Bradford, United Kingdom. Using this RMS, output-only vibration response data were acquired when the stand was empty and when it was occupied during football and rugby matches. For the occupied stadium, modal properties that corresponded to different crowd activities (sitting, standing and jumping) were also extracted. The configurations of the crowd occupation during sporting events were monitored by correlating vibration response data with synchronised video images. Modal parameter estimations were made on data that were acquired from several events, hence improving the statistical reliability of the estimates. The observations showed that seated crowd occupation led to a decrease in natural frequencies and an increase in damping ratios. Further decreases in natural frequencies were observed when the crowd stood and/or jumped following the scoring of a goal. These observations show that crowd occupation does not only contribute mass to structure, it acts as a dynamic system that interacts with the structure it is occupying. Furthermore, it has properties that vary according to the crowd configuration.
This paper presents a spatio-temporal 3D gait database and a view independent person identification method from gait. In case that a target changes one's walking direction compared with that in a database, the correct classification rate is reduced due to the appearance change. To deal with this problem, several methods based on a view transformation model, which converts walking images from one direction to virtual images from different viewpoints, have been proposed. However, the converted image may not coincide the real one, since the target is not included in the training dataset to obtain the transformation model. So we propose a view independent person identification method which creates a database with virtual images synthesized directly from the target's 3D model. In the proposed method, firstly we built a spatio-temporal 3D gait database using multiple cameras, which consists of sequential 3D models of multiple walking people. Then virtual images from multiple arbitrary viewpoints are synthesized from 3D models, and affine moment invariants are derived from virtual images as gait features. In the identification phase, images of a target who walks in an arbitrary direction are taken from one camera, and then gait features are calculated. Finally the person is identified and one's walking direction is estimated. Experiments using the spatio-temporal 3D gait database show the effectiveness of the proposed method.
In this paper, we present a multiple concurrent occupant identification approach through footstep-induced floor vibration sensing. Identification of human occupants is useful in a variety of indoor smart structure scenarios, with applications in building security, space allocation, and healthcare. Existing approaches leverage sensing modalities such as vision, acoustic, RF, and wearables, but are limited due to deployment constraints such as line-of-sight requirements, sensitivity to noise, dense sensor deployment, and requiring each walker to wear/carry a device. To overcome these restrictions, we use footstep-induced structural vibration sensing. Footstep-induced signals contain information about the occupants' unique gait characteristics, and propagate through the structural medium, which enables sparse and passive identification of indoor occupants. The primary research challenge is that multiple-person footstep-induced vibration responses are a mixture of structurally-codependent overlapping individual responses with unknown timing, spectral content, and mixing ratios. As such, it is difficult to determine which part of the signal corresponds to each occupant. We overcome this challenge through a recursive sparse representation approach based on cosine distance that identifies each occupant in a footstep event in the order that their signals are generated, reconstructs their portion of the signal, and removes it from the mixed response. By leveraging sparse representation, our approach can simultaneously identify and separate mixed/overlapping responses, and the use of the cosine distance error function reduces the influence of structural codependency on the multiple walkers' signals. In this way, we isolate and identify each of the multiple occupants' footstep responses. We evaluate our approach by conducting real-world walking experiments with three concurrent walkers and achieve an average F1 score for identifying all persons of 0.89 (1.3x baseline improvement), and with a 10-person "hybrid" dataset (simulated combination of single-walker real-world data), we identify 2, 3, and 4 concurrent walkers with a trace-level accuracy of 100%, 93%, and 73%, respectively, and observe as much as a 2.9x error reduction over a naive baseline approach.
The ability to identify pedestrians unobtrusively is essential for smart buildings to provide customized environments, energy saving, health monitoring and security-enhanced services. In this paper, we present an unobtrusive pedestrian identification system by passively listening to people's walking sounds. The proposed acoustic system can be easily integrated with the widely deployed voice assistant devices while providing the context awareness ability. This work focuses on two major tasks. Firstly, we address the challenge of recognizing footstep sounds in complex indoor scenarios by exploiting deep learning and the advanced stereo recording technology that is available on most voice assistant devices. We develop a Convolutional Neural Network-based algorithm and the footstep sound-oriented signal processing schemes to identify users by their footstep sounds accurately. Secondly, we design a "live" footstep detection approach to defend against replay attacks. By deriving the novel inter-footstep and intra-footstep characteristics, we distinguish live footstep sounds from the machine speaker's replay sounds based on their spatial variances. The system is evaluated under normal scenarios, traditional replay attacks and the advanced replays, which are designed to forge footstep sounds both acoustically and spatially. Extensive experiments show that our system identifies people with up to 94.9% accuracy in one footstep and shields 100% traditional replay attacks and up to 99% advanced replay attacks.
Occupant detection and recognition support functional goals such as security, healthcare, and energy management in buildings. Typical sensing approaches, such as smartphones and cameras, undermine the privacy of building occupants and inherently affect their behavior. To overcome these drawbacks, a non-intrusive technique using floor-vibration measurements, induced by human footsteps, is outlined. Detection of human-footstep impacts is an essential step to estimate the number of occupants, recognize their identities and provide an estimate of their probable locations. Detecting the presence of occupants on a floor is challenging due to ambient noise that may mask footstep-induced floor vibrations. Also, signals from multiple occupants walking simultaneously overlap, which may lead to inaccurate event separation. Signals corresponding to events, once extracted, can be used to identify the number of occupants and their locations. Spurious events such as door closing, chair dragging and falling objects may produce vibrations similar to footstep-impacts. Signals from such spurious events have to be discarded as outliers to prevent inaccurate interpretations of floor vibrations for occupant detection. Walking styles differ among occupants due to their anatomies, walking speed, shoe type, health and mood. Thus, footstep-impact vibrations from the same person may vary significantly, which adds uncertainty and complicates occupant recognition. In this paper, efficient strategies for event-detection and event-signal extraction have been described. These strategies are based on variations in standard deviations over time of measured signals (using a moving window) that have been filtered to contain only low-frequency components. Methods described in this paper for event detection and event-signal extraction perform better than existing threshold-based methods (fewer false positives and false negatives). Support vector machine classifiers are used successfully to distinguish footsteps from other events and to determine the number of occupants on a floor. Convolutional neural networks help recognize the identity of occupants using footstep-induced floor vibrations. The utility of these strategies for footstep-event detection, occupant counting, and recognition is validated successfully using two full-scale case studies.
Person identification plays a critical role in a large range of applications. Recently, RF based person identification becomes a hot research topic due to the contact-free nature of RF sensing that is particularly appealing in current COVID-19 pandemic. However, existing systems still have multiple limitations: i) heavily rely on the gait patterns of users for identification; ii) require a large amount of data to train the model and also extensive retraining for new users and iii) require a large frequency bandwidth which is not available on most commodity RF devices for static person identification. This paper proposes RF-Identity, an RFID-based identification system to address the above limitations and the contribution is threefold. First, by integrating walking pattern features with unique body shape features (e.g., height), RF-Identity achieves a high accuracy in person identification. Second, RF-Identity develops a data augmentation scheme to expand the size of the training data set, thus reducing the human effort in data collection. Third, RF-Identity utilizes the tag diversity in spatial domain to identify static users without a need of large frequency bandwidth. Extensive experiments show an identification accuracy of 94.2% and 95.9% for 50 dynamic and static users, respectively.
Current sensor technologies enable the passive and continuous monitoring of human behaviors as well as infrastructures to ensure personal safety and assess individual health state. One passive technology that has the potential of gathering personal data is the vibration sensor. In this paper, we carry out an extensive survey of the current vibration-based sensing technologies for human and infrastructure safety as well as health monitoring. These technologies utilize structural and body vibration as a source of data, and they can be incorporated into wearable or non-wearable devices. Furthermore, the vibration sensing technology utilizes low-cost and low-power sensors, which make it attractive for indoor and outdoor monitoring. We have classified the technologies into five categories: vibration-based sensing for assessing human health, recognizing personal behavior, inferring occupancy information, evaluating personal safety, and monitoring infrastructure health. In each category, we also classify the approaches that utilize single and multiple sensors. Moreover, we discuss the different types of signal processing and machine learning techniques that are applied to each approach.
People learn throughout life. However, incrementally updating conventional neural networks leads to catastrophic forgetting. A common remedy is replay, which is inspired by how the brain consolidates memory. Replay involves fine-tuning a network on a mixture of new and old instances. While there is neuroscientific evidence that the brain replays compressed memories, existing methods for convolutional networks replay raw images. Here, we propose REMIND, a brain-inspired approach that enables efficient replay with compressed representations. REMIND is trained in an online manner, meaning it learns one example at a time, which is closer to how humans learn. Under the same constraints, REMIND outperforms other methods for incremental class learning on the ImageNet ILSVRC-2012 dataset. We probe REMIND’s robustness to data ordering schemes known to induce catastrophic forgetting. We demonstrate REMIND’s generality by pioneering online learning for Visual Question Answering (VQA) (https://github.com/tyler-hayes/REMIND).
This paper presents a floor-vibration-based step-level occupant-detection approach that enables detection across different structures through model transfer. Detecting the occupants through detecting their footsteps (i.e., step-level occupant detection) is useful in various smart building applications such as senior/healthcare and energy management. Current sensing approaches (e.g., vision-based, pressure-based, radio frequency–based, and mobile-based) for step-level occupant detection are limited due to installation and maintenance requirements such as dense deployment and requiring the occupants to carry a device. To overcome these requirements, previous research used ambient structural vibration sensing for footstep modeling and step-level occupant detection together with supervised learning to train a footstep model to distinguish footsteps from nonfootsteps using a set of labeled data. However, floor-vibration-based footstep models are influenced by the structural properties, which may vary from structure to structure. Consequently, a footstep model in one structure does not accurately capture the responses in another structure, which leads to high detection errors and the costly need for acquiring labeled data in every structure. To address this challenge, the effect of the structure on the footstep-induced floor vibration responses is here characterized to develop a physics-driven model transfer approach that enables step-level occupant detection across structures. Specifically, the proposed model transfer approach projects the data into a feature space in which the structural effects are minimized. By minimizing the structure effect in this projected feature space, the footstep models mainly represent the differences in the excitation types and therefore are transferable across structures. To this end, it is analytically shown that the structural effects are correlated to the maximum-mean-discrepancy (MMD) distance between the source and target marginal data distributions. Therefore, to reduce the structural effect, the MMD between the distributions in the source and target structures is minimized. The robustness of the proposed approach was evaluated through field experiments in three types of structures. The evaluation consists of training a footstep model in a set of structures and testing it in a different structure. Across the three structures, the evaluation results show footstep detection F1 score of up to 99% for the proposed approach, corresponding to a 29-fold improvement compared to the baseline approach, which do not transfer the model.
The study of the human‐structure interaction (HSI) using biodynamics models has gained attention lately. Several studies have demonstrated that the passive (standing still) and active (bobbing/bouncing or walking) persons can act for the benefit of the structural system by considering their body dynamic properties. Nevertheless, little concern has been addressed regarding the HSI during jumping loads on floors. This kind of human load is often considered as a “force‐only” model by design guides, and the body dynamics is disregarded. Therefore, aiming at filling this gap, this work investigates experimental and numerically an individual jumping on a vibrating (flexible) floor mounted in the laboratory. The active HSI was evaluated considering both single and two degree of freedom models in time and frequency domains. Besides, the assessment of the human body dynamic parameters (spring, mass and damper) was carried out based on optimisation techniques. The results show the potential benefit of taking into account the active HSI in near‐resonant cases to the detriment of a force‐only model.
Structural vibration-based human sensing provides an alternative approach for device-free human monitoring, which is used for healthcare, space and energy usage management, etc. Prior work on this approach mainly focused on one person walking scenarios, which limits their widespread application. The challenge with multiple walkers is that the observed vibration response is a mixture of each walker's footstep-induced response, and it is difficult to identify 1) how many concurrent walkers are present, and 2) the timing of their footstep impacts on the floor. As a result, the extraction of detailed location information for each walker is erroneous. To address this challenge, we propose a structure-informed vibration signal characterization method to enable the detection and localization of overlapping vibration signals induced by multiple concurrent walkers. The intuition is that, due to the randomness in people's behavior, their footsteps do not impact the floor exactly at the same time and overlap partially. We decompose the signal to a non-fundamental frequency band which contains the heel strike onset information. With this decomposed signal, we can identify the number of walkers and use the initial peak information to localize each person independently. We conducted real-world experiments with up to three concurrent walkers and our system achieved a detection rate of up to 90% and an average localization error of 0.65m (2.9X baseline improvement).
Occupant identification proves crucial in many smart home applications such as automated home control and activity recognition. Previous solutions are limited in terms of deployment costs, identification accuracy, or usability. We propose SenseTribute, a novel occupant identification solution that makes use of existing and prevalent on-object sensors that are originally designed to monitor the status of objects to which they are attached. SenseTribute extracts richer information content from such on-object sensors and analyzes the data to accurately identify the person interacting with the objects. This approach is based on the physical phenomenon that different occupants interact with objects in different ways. Moreover, SenseTribute may not rely on users’ true identities, so the approach works even without labeled training data. However, resolution of information from a single on-object sensor may not be sufficient to differentiate occupants, which may lead to errors in identification. To overcome this problem, SenseTribute operates over a sequence of events within a user activity, leveraging recent work on activity segmentation. We evaluate SenseTribute using real-world experiments by deploying sensors on five distinct objects in a kitchen and inviting participants to interact with the objects. We demonstrate that SenseTribute can correctly identify occupants in 96% of trials without labeled training data, while per-sensor identification yields only 74% accuracy even with training data.
The scientific literature on automated gait analysis for human recognition has grown dramatically over the past 15 years. A number of sensing modalities including those based on vision, sound, pressure, and accelerometry have been used to capture gait information. For each of these modalities, a number of methods have been developed to extract and compare human gait information, resulting in different sets of features. This paper provides an extensive overview of the various types of features that have been utilized for each sensing modality and their relationship to the appearance and biomechanics of gait. The features considered in this work include (a) static and dynamic (temporal) features; (b) model-based and model-free visual features; (c) ground reaction force-based and finely resolved underfoot pressure features; (d) wearable sensor features; and (e) acoustic features. We also review the factors that impact gait recognition, and discuss recent work on gait spoofing and obfuscation. Finally, we enumerate the challenges and open problems in the field of gait recognition.
Many human activities induce excitations on ambient structures with various objects, causing the structures to vibrate. Accurate vibration excitation source detection and characterization enable human activity information inference, hence allowing human activity monitoring for various smart building applications. By utilizing structural vibrations, we can achieve sparse and non-intrusive sensing, unlike pressure- and vision-based methods. Many approaches have been presented on vibration-based source characterization, and they often either focus on one excitation type or have limited performance due to the dispersion and attenuation effects of the structures. In this paper, we present our method to characterize two main types of excitations induced by human activities (impulse and slip-pulse) on multiple structures. By understanding the physical properties of waves and their propagation, the system can achieve accurate excitation tracking on different structures without large-scale labeled training data. Specifically, our algorithm takes properties of surface waves generated by impulse and of body waves generated by slip-pulse into account to handle the dispersion and attenuation effects when different types of excitations happen on various structures. We then evaluate the algorithm through multiple scenarios. Our method achieves up to a six times improvement in impulse localization accuracy and a three times improvement in slip-pulse trajectory length estimation compared to existing methods that do not take wave properties into account.
Device-free human sensing is a key technology to support many applications such as indoor navigation and activity recognition. By exploiting WiFi signals reflected by human body, there have been many WiFi-based device-free human sensing applications. Among these applications, person identification is a fundamental technology to enable user-specific services. In this paper, we present Rapid, a system that can perform robust person identification in a device-free and low-cost manner, using fine-grained channel information (i.e., CSI) of WiFi and acoustic information from footstep sound. In order to achieve high accuracy in real-life scenarios with both system and environment noise, we perform noise estimation and include two different confidence values to quantify the impact of noise to both CSI and acoustic measurements. Based on an accurate gait analysis, we then adaptively fuse CSI and acoustic measurements to achieve robust person identification. We implement low-cost Rapid nodes and evaluate our system using experiments at multiple locations with a total of 1800 gait instances from 20 volunteers, and the results show that Rapid identifies a subject with an average accuracy of 92% to 82% from a group of 2 to 6 subjects, respectively.
Over the last decade or two, digital micro-electro-mechanical system (MEMS) accelerometers have been proposed as a replacement for analog electromagnetic coiled geophones. Although many positive results have been reported in articles and at conferences, other authors have claimed that there are no obvious differences between them. This article compares the analog geophone and MEMS accelerometer and our results show that the MEMS accelerometer has made some improvements on electric specifications. However, these improvements are so slight that identifying the enhanced signals obtained by the MEMS accelerometer is extremely challenging, if not impossible. Whether these improvements can evolve into significant acquisition benefits depends on many other factors involved in a seismic survey and which may contaminate the weak signal and effectively cancel most of the accelerometer's improvement.
Footbridges start to sway when packed with pedestrians falling into step with their vibrations.
In this paper, we present a room-level building occupancy estimation system (BOES) utilizing low-resolution
vibration sensors that are sparsely distributed. Many ubiquitous computing and building maintenance systems
require fine-grained occupancy knowledge to enable occupant centric services and optimize space and energy
utilization. The sensing infrastructure support for current occupancy estimation systems often requires multiple
intrusive sensors per room, resulting in systems that are both costly to deploy and difficult to maintain. To
address these shortcomings, we developed BOES. BOES utilizes sparse vibration sensors to track occupancy
levels and activities. Our system has three major components. 1) It extracts features that distinguish occupant
activities from noise prone ambient vibrations and detects human footsteps. 2) Using a sequence of footsteps, the
system localizes and tracks individuals by observing changes in the sequences. It uses this tracking information to
identify when an occupant leaves or enters a room. 3) The entering and leaving room information are combined
with detected individual location information to update the room-level occupancy state of the building. Through
validation experiments in two different buildings, our system was able to achieve 99.55% accuracy for event
detection, less than three feet average error for localization, and 85% accuracy in occupancy counting.
Human activity-induced vibrations in slender structural systems become apparent in many different excitation modes and consequent action effects that cause discomfort to occupants, crowd panic and damage to public infrastructure. Resulting loss of public confidence in safety of structures, economic losses, cost of retrofit and repairs can be significant. Advanced computational and visualisation techniques enable engineers and architects to evolve bold and innovative structural forms, very often without precedence. New composite and hybrid materials that are making their presence in structural systems lack historical evidence of satisfactory performance over anticipated design life. These structural systems are susceptible to multi-modal and coupled excitation that are very complex and have inadequate design guidance in the present codes and good practice guides. Many incidents of amplified resonant response have been reported in buildings, footbridges, stadia and other crowded structures with adverse consequences. As a result, attenuation of human-induced vibration of innovative and slender structural systems very often requires special studies during the design process. Dynamic activities possess variable characteristics and thereby induce complex responses in structures that are sensitive to parametric variations. Rigorous analytical techniques are available for investigation of such complex actions and responses to produce acceptable performance in structural systems.
This paper presents an overview and a critique of existing code provisions for human-induced vibration followed by studies on the performance of three contrasting structural systems that exhibit complex vibration. The dynamic responses of these systems under human-induced vibrations have been carried out using experimentally validated computer simulation techniques. The outcomes of these studies will have engineering applications for safe and sustainable structures and a basis for developing design guidance.
Suppose we are given a learning set \(\mathcal{L}\) of multivariate observations (i.e., input values \(\mathfrak{R}^r\)), and suppose each observation is known to have come from one of K predefined classes having similar characteristics. These classes may be identified, for example, as species of plants, levels of credit worthiness of customers, presence or absence of a specific medical condition, different types of tumors, views on Internet censorship, or whether an e-mail message is spam or non-spam.
This paper considers the person verification problem in modern surveillance and video retrieval systems. The problem is to identify whether a pair of face or human body images is about the same person, even if the person is not seen before. Traditional methods usually look for a distance (or similarity) measure between images (e.g., by metric learning algorithms), and make decisions based on a fixed threshold. We show that this is nevertheless insufficient and sub-optimal for the verification problem. This paper proposes to learn a decision function for verification that can be viewed as a joint model of a distance metric and a locally adaptive thresholding rule. We further formulate the inference on our decision function as a second-order large-margin regularization problem, and provide an efficient algorithm in its dual from. We evaluate our algorithm on both human body verification and face verification problems. Our method outperforms not only the classical metric learning algorithm including LMNN and ITML, but also the state-of-the-art in the computer vision community.
The London Millennium Footbridge is located across the Thames River in Central London. At its opening on June 10, 2000, the bridge experienced pedestrian-induced lateral vibration. Observations on the day of opening and studies of video footage revealed up to 50 mm of lateral movement of the south span and 70 mm of the center span. The north span did not move substantially. The bridge was closed on June 12, 2000, pending an investigation into the cause of the unexpected lateral movements. This paper highlights the phenomenon of pedestrian-induced lateral vibration on footbridges and the current state of knowledge of the lateral loading effect. Modification of the bridge, introducing extensive passive damping, is currently underway with completion scheduled for the end of 2001.
In this paper, a novel view invariant person identification method based on human activity information is proposed. Unlike most methods proposed in the literature, in which walk (i.e., gait) is assumed to be the only activity exploited for person identification, we incorporate several activities in order to identify a person. A multicamera setup is used to capture the human body from different viewing angles. Fuzzy Vector Quantization and Linear Discriminant Analysis are exploited in order to provide a discriminant activity representation. Person identification, activity recognition and viewing angle specification results are obtained for all the available cameras independently. By properly combining these results, a view-invariant activity-independent person identification method is obtained. The proposed approach has been tested in challenging problem setups, simulating real application situations. Experimental results are very promising.
This paper presents methods for footstep-based person identification using a large pressure-sensitive floor with a sensory system. The aim was to analyse and compare different pattern classification methods for their ability to solve this particular problem as well as to introduce some novel and useful methodological extensions, which can improve classification accuracy and the adaptability of the system. These extensions are based on the conditional posterior probability outputs of classifiers, i.e., efforts to combine classifiers trained with different feature sets and to combine multiple footstep instances of a single person walking on the floor. Additionally, a method to reject unreliable examples in order to increase accuracy was applied to the system. The experiments demonstrated the usefulness of these methods. An identification method that uses a combination of multiple classifiers and multiple examples yielded very promising results with an overall accuracy rate of 92% for ten different walkers. When the reject option was added, a classification rate of 95% with a 9% rejection rate was achieved. This methodology can be applied to smart room applications where a small number of persons need to be identified.
Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.