In the era of smart machines, even real time insights are late. Today, predictive and prescriptive are the operative words. While, human expertise can perceive symptoms of disorder in systems – men, materials, or machines – to an extent, it is subjective and inconsistent, and more importantly, usually occurs after a certain amount of damage is already done. Simply put, human perception is just not “predictive” enough.
Sensor fitted machinery, generates vast amounts of data which often goes to corroborate observations from human perception. But curiously enough, there are still spaces that neither human perception nor sensors can reach easily. These spaces have been of special interest to our team. In this article we have highlighted how we enable better and more reliable decision making to interpret health of systems.
Download the full article or click the button to continue reading.
In the era of smart machines, even real time insights are late. Today, predictive and prescriptive are the operative words. Human expertise can perceive symptoms of disorder in systems – men, materials, or machines – to an extent. A plant operator can sense an anomaly in the vibration signature of, say, rolling element bearings in a gear transmission system or sniff out the distinct rotten-egg smell that signifies a leak or oxidation or corrosion in machinery. A food taster can sample a dish to determine its quality and taste. A doctor can understand a patient’s condition to a certain extent through observation. But human perception is subjective and inconsistent, and more importantly, usually occurs after a certain amount of damage is already done. Simply put, human perception is just not “predictive” enough.
Sensors, which take the form of tiny electronic pieces, have been changing the game over the last few decades. The machinery in new industrial setups is often equipped with sensors and legacy systems that are retrofitted with physical/soft sensors wherever possible. These sensors generate vast amounts of data which often goes to corroborate observations from human perception. Together, they offer keener insights.
But curiously enough, there are still spaces that neither human perception nor sensors can reach easily. These spaces have been of special interest to our team. Our vision is to enable human decision-makers to better interpret the health of systems, so that decision-making becomes easier, more reliable, and consistently effective.
The “Sensor Concept Map” (figure 1) provides an overview of sensing modalities and inferences drawn from the same (via the additional information they provide) for augmented human perception of humans, materials, and machines. The objective is to detect their activities/operational states/defects in their natural habitat, sense subtle variations in behavioral patterns, draw rich insights, and enable accurate predictive models.
Human beings are fundamentally interested in learning more about themselves. Health is a prime concern. Multiple sectors focus on human health: pharma, healthcare, insurance, and the government, to name a few. With sensors, organizations operating in these spheres can now help get a better view of our health, movements, and activities in order to predict disorder and prescribe remedies.
To understand human parameters as well as the human context, our team employs a variety of sensing mechanisms to observe human location, activity, and physiology among other relevant factors. For location sensing outdoors, we primarily rely on GPS or A-GPS. For indoor scenarios, we have actively worked on indoor localization using personal devices. Here we rely on Radio Frequency (RF) transceivers present in devices such as WiFi and Bluetooth Low Energy (BLE) peripherals. For indoor location sensing, we created a novel radiowave propagation path loss model which can locate a person with high precision. For activity recognition, we relied on inertial sensors like accelerometers and gyroscopes to understand human gestures, postures, and gait-based activities. We realized that the barometer sensor helps provide additional context to recognize activities like sitting, standing, or falling. We have also looked at physiological parameters like fatigue, heart-rate, breathing-rate, and the breathing power of individuals using wearable devices.
We wanted to obtain human parameters in sizeable detail while letting the subject lead her daily life in her natural setting, in order to remove the element of “observation bias” from human behavior. Here, it becomes clear that electromagnetic imaging-based human parameter elucidation is the future holy grail of human sensing. A step forward in this direction led us towards the design of small, reliable, energy-efficient and customizable (deployment-ready) millimeter wave (MMW) radar units for applications in our daily lives. Radar-based human sensing offers three distinct advantages:
The ability to sense at a distance
Preservation of privacy
Insulation from the effects of ambience (illumination, temperature, and fire among other factors)
Multiple small-sized, networked radar modules can be spread across an apartment so that the occupant’s location, movement, and physiology are seamlessly recorded at every moment.
We have used these low-level inferences from human sensing to infer a higher level of human knowledge, of which a few case studies are illustrated below. In this respect, we have envisaged a human sensing platform as shown in figure 2.
The human sensing platform allows us to use sensor data to draw low-level inferences related to location, activity, physiology, and cognition. This can be further augmented with sensor-driven insights into the psychological state (example GSR, EEG, ECG). Hence, when the psychology, location, and activity are correlated using an algorithm, it can yield a better model of the mathematics of human behavior. Domain adaptation is used for this purpose along with the knowledge base of human nature which is derived from research in behavioral sciences.
Spectral vibration analysis of an operation machine offers critical information about the machine’s current state. Rotating machines experience anomalies due to situations such as mass imbalance, mechanical wear and tear, and misalignments, among others. Such anomalies result in vibrations induced on non-rotating parts, noisy environments, increased harmonic distortions in power supply, and other undesirable outcomes. To address this challenge, we built a unique non-contact vibration measurement sensing system termed the “RF Guided Stroboscope”. Optical stroboscopes are traditionally used for the visual inspection of vibrating parts. Additionally, if stroboscopes could extract phase information from vibration signatures, it could lead to major possibilities in prognosis. However, such stroboscopes are not cost-effective in large deployment scenarios. We tackled this issue through a smart combination of the best of both worlds, namely the stroboscope and radar. In figure 3, the schematic is presented.
This system combines two separate sensing methodologies in a unified approach for inference. Initially, the Doppler radar captures the spectrum of all vibrations in its field of view (FOV). The set of deduced frequencies is then conveyed to the stroboscope. The stroboscope, in turn, illuminates the FOV using an LED panel, one frequency at a time, thereby enabling the identification of all the parts. This system is completely automated and significantly faster than a standard stroboscope, which needs to search for the unknown frequency. Coupled with a detailed spectral analysis of vibration (amplitude and frequency of harmonics), this low-cost and non-contact system offers an elegant solution for legacy machines as well as for extreme environments like high temperature operations.
Micro-defects in materials can cause major problems. This is clearly evident during production processes in manufacturing and fabrication. For a ‘zero-defect’ production process that ensures product integrity and reliability, defects need to be detected upstream of the actual process to prevent problems from snowballing later. Optical techniques such as infrared thermography, speckle imaging, digital holographic techniques, microscopy, and THz imaging aim to augment standard vision-based inspection approaches for inspections with high sensitivities and specificities. As part of our technology roadmap, we are putting together a portfolio of sensors comprising elements from optics, acoustics, and electronics. These elements are finely tuned for specific use cases in this context. Based on whether a large piece of machinery or a small additively manufactured component is being inspected, we use principles such as light scattering, spectroscopy, acoustics, and interferometry to develop rugged, cost-effective, versatile, multi-modal platforms.
Humans need technologies that help them perceive and interact in real time. This becomes possible when their natural communication channels are augmented with synthesized information. Audio and haptic augmentation systems form an interesting “non-graphical display” alternative to vision. We also propose to stretch the sensing paradigm from merely sensing location, activity, and physiology, to actually recognizing emotional states using emerging sensing paradigms like “Volatile Organic Compound (VOC)”, which is the biochemical state of the human subject, in order to perceive actionable insights on the subject using the domain of HII. Again, with the radar modules in place, a natural extension of this concept would be to build models of human-articulated motion like gait analysis, which takes us in the direction of “person identification”, a potential solution with many significant uses. Similarly, a configurable and high-precision ultra-wideband (UWB) radio frequency (RF) imaging system can act as a set of powerful ‘eyes’ in dynamic factory environment(s), thereby enabling worker safety applications. System reliability in a dynamic environment is ensured only when the proposed system automatically adapts itself to the changes unfolding in the scene. Possibilities include the study of human range-of-motion (ROM) and activity using smart fibers enabled with opto-electronic sensing systems. Next-generation vision may include compound bionic eyes which use microfluidics-based liquid lenses.
The future is near with exciting developments in “metamaterial”-based sensor designs as well as “quantum sensing”. Metamaterials (beyond materials) are artificially synthesized materials with unusual properties, not seen in our world. Since the beginning of this century, the subject of metamaterials has been intensely explored. However, we are yet to see genuine engineering applications. Metamaterial-based sensors will significantly enhance the performance of sensors while reducing their size. Quantum sensing allows us to reach the absolute limits of measurement resolution and sensitivity, and in doing so, extends our reach far beyond classical measurement strategies. Recent studies demonstrate the possibility of completely interaction-free measurements that are theoretically impossible according to classical sensing principles. Such interaction-free measurement techniques will eventually enable us to see the ‘unseen’.
Active Assisted Living
Active Assisted Living (AAL) projects have humans living independently or aided by a smart environment. We worked with these environments to monitor human subjects using ambient sensors and intelligence. However, AAL also brings in the paradigm of actuation, which in turn leads to new challenges in the domain. When a robot and a human coexist in proximity, the robot needs to have a deep behavioral understanding of the human to effectively interact with her. This becomes more challenging in AAL scenarios as the subjects (such as geriatric subjects) are often cognitively challenged.
The scope of such research work is to continuously, but non intrusively, monitor the subject using wearables and infrastructure-based sensors, and then make intelligent inferences on the wellness, safety, and behavioral parameters of the subject to create a deep model of the person. Such deep models can be used to study the wellness, cognition, and emotion of the human subject. In our current research, we concentrate on monitoring the elderly. However, the insights gained from the study can be generalized to other domains involving human factors and human understanding.
Driver Behavior Modeling
In this study, we try to understand the skill and aggression levels of a driver based on mobile sensor data. Every driver attempts to minimize risk through a psychologically induced “task difficulty” homeostasis—a state of equilibrium that is maintained by weighing out one’s own driving skill against the demands of driving under specified conditions. In the process of navigating the vehicle through a plethora of difficult road conditions, a driver’s innate ability as well as traffic psychology comes to the fore. This ability can be modeled by monitoring the driving pattern over long periods of time, thus giving rise to the driver’s unique behavioral profile. Driver behavior models have many practical uses like uniquely identifying the driver as well as greatly improving road safety by inducing safe driving styles.