Expert Insights
AI-TechScienceGovernmentBusinessSportsEntertainment
AI-Tech

Contactless heart monitoring improves when neural networks train on video signals

Teaching pulse-tracking software from facial video rather than finger sensors removes hardware delays and lowers dataset costs, according to a new computational benchmark.

By Primary News Team

18 statements added by Bhargav Acharya, lead author · 21 Sep see what was added
Contactless heart monitoring improves when neural networks train on video signals
Source: Noldus
Published21 Sep 2026, 21:07 Last updated21 Sep 2026, 21:07 Sources
Source mode Marks each sentence drawn from a source or a contributor

With every contraction of the heart, a fresh surge of oxygenated blood travels through arteries into the dense bed of capillaries beneath the skin. As blood volume rises and falls, the optical absorption of the tissue fluctuates in tandem. Facial skin absorbs slightly more light during each systolic surge, particularly within the green band of the visible spectrum where hemoglobin absorption peaks, and reflects slightly more light as pressure recedes.12 Standard digital cameras record these subtle variations across consecutive frames, registering rhythmic intensity shifts that remain invisible to the naked human eye.

Translating these microscopic color shifts into a dependable pulse reading is the objective of remote photoplethysmography, a contactless monitoring technique aimed at assessing vital signs without cables or physical attachments. The method holds substantial promise for telemedicine consultations, neonatal intensive care units, and continuous health tracking in assisted living facilities.12 In clinical environments where adhesive patches tear fragile skin or where patients cannot tolerate wearable clips, an ordinary camera lens can serve as a non-invasive diagnostic sensor. The technology offers a practical way to monitor cardiovascular function across populations without requiring specialised hardware distribution.

To convert raw video into an accurate pulse measurement, a system must follow a precise sequence of events. A camera first captures sequential video frames of a person under ambient illumination. Software detects the face, isolates regions of exposed skin, and tracks changes in pixel brightness across color channels over time. From these minute oscillations, an algorithm isolates the cardiac waveform from background noise caused by minor head movements or flickering room lights. Finally, the dominant frequency of that extracted waveform is converted into a standard heart rate measurement in beats per minute.

For years, the most accurate software for this task has relied on supervised deep learning, where neural networks learn to predict heart rate by comparing video recordings against pulse readings collected simultaneously from fingertip contact clips.12 A preprint posted to the arXiv repository by Bhargav Acharya, Barbara Hammer, and Hanna Drimalla at the Center for Cognitive Interaction Technology at Bielefeld University reports that this standard training convention introduces fundamental errors.1 The authors found that training neural networks on pseudo-labels, artificial reference signals generated automatically from the facial videos using classical signal-processing mathematics, matched or exceeded the performance of models trained on physical finger sensors.1

Why do conventional training datasets struggle with remote heart rate monitoring?

Conventional training datasets struggle because physical contact sensors placed on the hand do not measure the same physiological signal that appears across the human face. A pulse wave generated by the left ventricle does not reach the forehead and the fingertips at the same instant. As blood travels down the arterial tree toward the extremities, the pressure wave encounters vascular resistance, creating a measurable temporal delay known as pulse transit time.1 Capillary beds in the fingers also possess distinct vessel structures and tissue elasticity compared to facial microvasculature, producing different wave shapes and phase shifts.1

Acharya, Hammer, and Drimalla explain that this mismatch forces deep neural networks to learn an unnatural mapping during supervised training.1 When a model inspects video of a subject's face, the optical changes reflect blood arrival in facial capillaries, yet the ground-truth target signal recorded from the finger lags behind. Bhargav Acharya told Primary that under ideal conditions, "you would want a one-to-one mapping, where each frame of the video maps directly to the true facial pulse waveform."contributed Acharya explained that because "the phase shift between the facial pulse and the finger signal varies from participant to participant," the machine learning model is forced to correlate facial video against "a finger signal that is differently shifted for each individual," which "can prevent the visual layers from learning the true temporal characteristics of the facial pulse."contributed Previous research cited by the authors showed that training neural networks on contact sensors mounted directly on the forehead reduced mean squared error against true waveforms by up to 40% compared to finger-mounted devices.1 However, affixing physical monitors to the face obscures skin features, disrupts natural facial expressions, and makes realistic dataset collection difficult.

Contactless heart monitoring improves when neural networks train on video signals
Source: Quinn Dombrowski from Berkeley, USA (CC BY-SA 2.0)

Temporal misalignment between recording devices compounds the biological delay. Standard photoplethysmography datasets rely on separate hardware components: a camera filming video frames and an independent medical monitor digitizing fingertip pulse waves at high frequencies.12 Acharya, Hammer, and Drimalla state that establishing millisecond-level synchronization across independent devices requires dedicated hardware triggers. In the majority of public research datasets, researchers rely instead on software clocks, which frequently drift or suffer variable communication latencies. The authors report that this lack of precise alignment, combined with pulse transit delays, degrades the training process of supervised deep learning models.1

How can algorithms generate their own training signals from video?

Algorithms generate their own training signals by applying deterministic optical equations to facial pixels to calculate blood volume changes without human labels or contact devices.12 This unsupervised approach relies on established skin-reflectance physics, which models how light interacts with melanin, hemoglobin, and surface tissue.12 Instead of requiring a neural network to deduce cardiovascular patterns from scratch, classical signal-processing methods extract periodic pulse signals directly from raw red, green, and blue color channels across cropped facial regions.

In their investigation, Acharya, Hammer, and Drimalla adopted a classical algorithm called Plane-Orthogonal-to-Skin, known as POS. The method defines a mathematical projection plane orthogonal to skin-tone variations in normalized color space, isolating pulse-induced color shifts while filtering out intensity changes caused by head motion. The authors cropped facial regions using an automated detection tool called YOLO5Face, extracted the raw optical signal using the POS formulation, applied a second-order Butterworth bandpass filter between 0.7 and 3.0 hertz, and computed the envelope of the signal using the Hilbert transform.1 The resulting waveform served as a training pseudo-label, generated entirely from the video itself.1

Acharya told Primary that the frequency range of 0.7 to 3.0 hertz was selected because it "corresponds to approximately 42–180 beats per minute, which we assume covers the plausible heart-rate range of the participants."contributed Acharya noted that the bandpass filter isolates frequency components within that physiological bracket, observing that "pseudo-label quality itself is not particularly sensitive to the exact boundaries, as long as they encompass the true heart rate."contributed However, Acharya cautioned that "if a participant’s actual heart rate falls outside this range, the corresponding signal will be attenuated or removed by the filter."contributed

The Bielefeld team systematically compared three training signals across their experiments.1 The first signal, designated Finger-PPG, used raw pulse waveforms collected simultaneously from fingertip contact devices at a sampling rate of 1,000 hertz, representing the conventional training approach.1 Describing the experimental configuration to Primary, Acharya stated that "the third training signal was Synced-PPG," adding that "it is the same waveform as the Finger-PPG signal, but temporally shifted to better align with the true facial pulse waveform."contributed That alignment was calculated by maximizing the cross-correlation with the facial POS signal, compensating for hardware latency and physiological transit delays while retaining the finger sensor's physical wave shape. The third signal, Pseudo-PPG, discarded the contact sensor entirely, using only the filtered POS waveform extracted from facial pixels.1

Acharya, Hammer, and Drimalla trained two established neural network architectures to isolate the influence of the training signal from structural model innovations. They selected TS-CAN, a convolutional network using temporal shift modules and attention mechanisms to process 160-frame video clips, alongside Physnet, a 3D convolutional encoder-decoder network processing 128-frame clips.1 All models were trained for 30 epochs with a batch size of 64 using the SALSA optimizer and a basic mean squared error loss function, avoiding complex contrastive learning frameworks to provide a direct assessment of label quality.1

When does algorithmic pseudo-labeling outperform physical sensors?

Algorithmic pseudo-labeling outperforms physical contact sensors when experimental datasets suffer from imperfect hardware synchronization between cameras and physiological monitors.1 The authors tested their framework on two public benchmarks with contrasting technical setups: the CHILL dataset and the PURE dataset.1 CHILL contains recordings from 45 participants across resting and elevated heart rate conditions under varied indoor lighting, relying on software-based synchronization between its camera and contact sensors.1 PURE features 10 participants performing systematic head movements, recorded using custom hardware that enforced strict temporal alignment.12

Contactless heart monitoring improves when neural networks train on video signals
Source: Kaggle

On the CHILL dataset, where software clocks introduced variable synchronization errors, models trained on video-derived pseudo-labels consistently outperformed models trained on the original finger sensors.1 When Acharya, Hammer, and Drimalla conducted participant-level 10-fold cross-validation, the non-parametric Wilcoxon signed-rank test confirmed statistically significant reductions in mean absolute error for both TS-CAN and Physnet when trained on Pseudo-PPG compared to Finger-PPG. Because the pseudo-label is derived directly from the video frames, its temporal phase aligns naturally with the visual signal, eliminating the timing discrepancies that disrupt supervised training.

On the PURE dataset, where external hardware maintained synchronization, the results showed a different pattern.1 In within-dataset evaluations, the authors found no statistically significant difference between models trained on Pseudo-PPG and those trained on contact sensors.1 For instance, Physnet trained on Pseudo-PPG achieved a median absolute error of 0.55 beats per minute across participants, compared to 0.62 beats per minute when trained on Finger-PPG and 0.67 beats per minute on Synced-PPG. In cross-dataset evaluations, where models trained on PURE were tested on CHILL, supervised models trained on physical sensors initially demonstrated superior generalization.1 However, the authors report that removing a single outlier participant whose video yielded noisy pseudo-labels significantly narrowed that performance gap, demonstrating that automated supervision depends heavily on baseline video quality.1

Addressing that discrepancy, Acharya told Primary that "for this participant, POS failed to extract a reliable estimate of the facial pulse waveform."contributed Because the classical extraction algorithm is susceptible to adverse recording conditions, "the resulting unreliable pseudo-label was used as a training target and contributed to the drop in performance," which Acharya noted "highlights the importance of assessing pseudo-label quality and filtering unreliable samples before using them for training."contributed Regarding the overall utility of the approach, Acharya stated in response to questions from Primary that "the key advantage is that pseudo-labels eliminate the need to collect contact-sensor data in the first place," emphasizing that even if hardware-synchronized finger sensors provide clean labels, they require a laborious collection process that "can potentially be avoided."contributed

The study is a computational benchmark conducted on two curated datasets comprising 55 total research participants recorded in controlled laboratory settings, meaning it cannot predict performance across diverse clinical populations, irregular cardiac rhythms, or extreme ambient lighting.contributed The paper was posted as a preprint to the arXiv repository and has not yet undergone formal peer review.1 Furthermore, the pseudo-labeling technique relies on the initial quality of the POS algorithm, which can degrade when subjects exhibit sudden, violent head movements or when camera sensors suffer from severe sensor noise.1 In response to questions from Primary about operational boundaries, Acharya warned against assuming broad generalization, saying, "I would be cautious about relying on the model in any clinical or physiological condition that was not represented in the training data."contributed Acharya noted that unrepresented factors include "very high or low heart rates, irregular heart rhythms, or conditions that substantially alter skin perfusion," advising that systems require targeted validation across the specific hardware and demographic groups encountered in clinical practice.contributed

What does contactless training mean for clinical telemedicine?

Contactless training means medical institutions and software developers can build robust computer vision diagnostics without staging expensive clinical trials solely to collect synchronized contact-sensor data. Gathering medical-grade reference data from patients requires calibrated hardware, technical staff, and participant consent forms, restricting the size and geographic variety of current training libraries.1 If standard digital video can provide its own reliable training labels, researchers can train neural networks on larger and more varied video collections, encompassing diverse skin complexions, age groups, and environmental settings.1

The findings also change how engineers approach sensor fusion in medical software. When hardware synchronization is absent or unreliable, forcing a deep neural network to match a delayed peripheral sensor can actively degrade the model's accuracy. By demonstrating that mathematically grounded signal processing can supply valid training supervisory signals, the Bielefeld research establishes a practical bridge between classical physiological modeling and modern machine learning.

The next analytical test will involve deploying pseudo-labeled neural networks on continuous, real-world telemedicine video streams where illumination changes dynamically and subjects speak or gesture freely. Researchers must determine whether automated labeling strategies can accommodate arrhythmias, where pulse wave intervals fluctuate irregularly from beat to beat. For now, the demonstration that video frames can teach neural networks without physical finger clips removes a persistent barrier to scaling contactless cardiac monitoring.

What this rests on

45 sentences trace to 2 sources and 1 contributor.

  1. 1 BA Bhargav Acharya, Barbara Hammer, Hanna Drimalla Paper · arxiv.org · 10 Sep 2026 Beyond Contact Sensors: Deep learning with Pseudo-Labeling for remote Photoplethysmography Preprint · may not have been peer reviewed See the source
  2. 2 A arxiv.org Paper · arxiv.org rPPG-Toolbox: Deep Remote PPG Toolbox Preprint · may not have been peer reviewed See the source
  3. 3 BA Bhargav Acharya, lead author Contributor · interview 21 Sep 2026 · in the article since 21 Sep 2026 Contribution — Bhargav Acharya 18 statements added to this article

Article history

  1. Published 21 Sep 2026, 21:07
    Assembled by the Primary desk from 2 sources · 1 contributor · 45 cited sentences
  2. Bhargav Acharya, lead author 18 statements added the day before publication
    1. Go to the sentence: Acharya, Hammer, and Drimalla explain that this mismatch forces deep neural networks to learn an unnatural mapping during supervised training.
    2. Go to the sentence: Bhargav Acharya told Primary that under ideal conditions, "you would want a one-to-one mapping, where each frame of the video maps directly to the true facial pulse waveform."
    3. Go to the sentence: Acharya explained that because "the phase shift between the facial pulse and the finger signal varies from participant to participant," the machine learning model is forced to correlate facial video against "a finger signal that is differently shifted for each individual," which "can prevent the visual layers from learning the true temporal characteristics of the facial pulse."
    4. Go to the sentence: The authors cropped facial regions using an automated detection tool called YOLO5Face, extracted the raw optical signal using the POS formulation, applied a second-order Butterworth bandpass filter between 0.7 and 3.0 hertz, and computed the envelope of the signal using the Hilbert transform.
    5. Go to the sentence: Acharya told Primary that the frequency range of 0.7 to 3.0 hertz was selected because it "corresponds to approximately 42–180 beats per minute, which we assume covers the plausible heart-rate range of the participants."
    6. Go to the sentence: Acharya noted that the bandpass filter isolates frequency components within that physiological bracket, observing that "pseudo-label quality itself is not particularly sensitive to the exact boundaries, as long as they encompass the true heart rate."
    7. Go to the sentence: However, Acharya cautioned that "if a participant’s actual heart rate falls outside this range, the corresponding signal will be attenuated or removed by the filter."
    8. Go to the sentence: The Bielefeld team systematically compared three training signals across their experiments.
    9. Go to the sentence: Describing the experimental configuration to Primary, Acharya stated that "the third training signal was Synced-PPG," adding that "it is the same waveform as the Finger-PPG signal, but temporally shifted to better align with the true facial pulse waveform."
    10. Go to the sentence: Algorithmic pseudo-labeling outperforms physical contact sensors when experimental datasets suffer from imperfect hardware synchronization between cameras and physiological monitors.
    11. Go to the sentence: However, the authors report that removing a single outlier participant whose video yielded noisy pseudo-labels significantly narrowed that performance gap, demonstrating that automated supervision depends heavily on baseline video quality.
    12. Go to the sentence: Addressing that discrepancy, Acharya told Primary that "for this participant, POS failed to extract a reliable estimate of the facial pulse waveform."
    13. Go to the sentence: Because the classical extraction algorithm is susceptible to adverse recording conditions, "the resulting unreliable pseudo-label was used as a training target and contributed to the drop in performance," which Acharya noted "highlights the importance of assessing pseudo-label quality and filtering unreliable samples before using them for training."
    14. Go to the sentence: Regarding the overall utility of the approach, Acharya stated in response to questions from Primary that "the key advantage is that pseudo-labels eliminate the need to collect contact-sensor data in the first place," emphasizing that even if hardware-synchronized finger sensors provide clean labels, they require a laborious collection process that "can potentially be avoided."
    15. Go to the sentence: The study is a computational benchmark conducted on two curated datasets comprising 55 total research participants recorded in controlled laboratory settings, meaning it cannot predict performance across diverse clinical populations, irregular cardiac rhythms, or extreme ambient lighting.
    16. Go to the sentence: Furthermore, the pseudo-labeling technique relies on the initial quality of the POS algorithm, which can degrade when subjects exhibit sudden, violent head movements or when camera sensors suffer from severe sensor noise.
    17. Go to the sentence: In response to questions from Primary about operational boundaries, Acharya warned against assuming broad generalization, saying, "I would be cautious about relying on the model in any clinical or physiological condition that was not represented in the training data."
    18. Go to the sentence: Acharya noted that unrepresented factors include "very high or low heart rates, irregular heart rhythms, or conditions that substantially alter skin perfusion," advising that systems require targeted validation across the specific hardware and demographic groups encountered in clinical practice.