Cognitive Modelling

PINNs, gaze prediction, and eye movements as signals of mental demand.

I treat the eye as an instrument. Gaze position, fixation patterns, blink dynamics, and pupil response are all measurable signals, and my research asks how much we can infer from them about attention and cognitive load — and how well we can model and predict them.

Coming from physics, I lean on physics-informed approaches. Eye movement is one of the rare human behaviours with a genuine mechanical description: the eye is a near-perfect sphere rotating about a fixed centre, and Donders’ and Listing’s laws constrain which orientations it can actually take. That makes it an unusually good candidate for physics-informed neural networks (PINNs), which are widely used in the natural sciences but rarely applied in HCI — largely because most HCI problems have no defensible governing equations. Eye tracking does.

Methods & tools: Python, PyTorch, physics-informed neural networks, Tobii Pro Spectrum & Pro Lab, SHAP, scikit-learn, MNE.

Physics-Informed Gaze Prediction

The core question: does constraining a neural network with the physics of eye rotation actually make it predict gaze better than the same network without those constraints? (Roumpas et al., 2025)

I derived partial differential equations describing eye and head movement in spherical coordinates, and added them as a physics loss term alongside the standard data loss. The comparison was deliberately tight — identical architecture (three fully connected layers, ReLU, two output branches for gaze points and angular dynamics), identical features, identical data — so the only difference was the physics. Data came from an adaptive-interface experiment with pilots, which gave a useful mix of steady fixations and rapid glances.

The PINN won, and won asymmetrically. It reached a mean absolute error of 0.61 horizontally and 0.35 vertically, and explained substantially more variance in the vertical axis (R²ᵧ = 0.91 vs 0.85 for the baseline). Vertical gaze is where the physical constraint bites hardest: MAE dropped 29% and RMSE 20%. Horizontal predictions improved on error terms but the two models stayed close on R², which is an honest limitation rather than a footnote — the horizontal constraint is doing less work than the vertical one.

A SHAP analysis explained why. Directional gaze features dominate, left/right vertical directions are strongly redundant with each other, and pupil diameter contributes almost nothing to gaze estimation — useful negative evidence for anyone building a feature set. Getting the physics loss to train stably needed adaptive loss scaling and gradient clipping; an unweighted physics term simply overwhelms the data term.

Which Architecture Actually Fits Eye Movement?

The obvious follow-up: PINNs beat a plain feedforward network, but gaze is a sequence on a graph of successive samples. Surely a temporal or relational architecture does better? (Roumpas & Xenos, 2026)

I tested that directly, moving to wearable eye-tracking glasses and comparing six models — a feedforward NN, a Graph Neural Network (each gaze sample a node, edges along temporal adjacency), a Temporal Convolutional Network, and physics-informed hybrids of all three.

The answer was no, and cleanly so. The pure PINN outperformed every alternative, reaching MAE 0.735 (X) and 0.716 (Y) with R² of 0.996 on both axes; paired Wilcoxon signed-rank tests put every comparison against it well below p < 0.01. The GNN and TCN landed at or below the plain baseline, and — the more interesting result — the PINN-GNN and PINN-TCN hybrids showed higher error variance than the pure PINN. Adding graph and temporal machinery on top of the physics made the model less stable, not more expressive. Trajectory plots show the same story qualitatively: the PINN tracks ground truth closely while the data-driven models produce visibly noisy paths, with the residual errors concentrated at screen edges where calibration is weakest anyway.

For HCI this is a practical finding. The cheapest model in the comparison is also the best one, provided you spend the effort on the equations instead of the architecture.

Blink Dynamics as a Discrete Signal

Gaze is continuous; blinks are events. Separating a deliberate blink from a reflexive one turns the eyelid into a usable input channel — but only if the separation is reliable. (Roumpas et al., 2025)

Working from eye-openness time series rather than pupil-loss detection (a more direct measure of eyelid movement), I collected recordings from 44 participants across blink, non-blink, and full-recording conditions, and trained a feedforward classifier on blink duration plus per-eye eye-openness means and standard deviations. It reaches 92% test accuracy.

The distributions themselves were the more interesting result. Voluntary blinks form a single broad peak of longer closures — the signature of controlled, intentional movement. Involuntary blinks are bimodal, showing two distinct peaks that survived a refined cluster-based reclassification. Whether that reflects genuine physiological subtypes, an artefact of the experimental conditions, or something else is still open, and it is the kind of loose end I would rather flag than paper over. The dataset and trained model are openly published so others can check it.

Where This Is Going

The three strands converge on one question: how far can oculomotor behaviour be read as a proxy for mental effort? A model accurate enough to predict where the eye should go makes its own failures informative — the moments where behaviour departs from the physical model are exactly the moments worth examining. Turning that into a validated marker of cognitive state, robust enough to drive an adaptive interface in real time, is the direction of my current work.

References

2026

Modelling Eye Mechanics Using Eye-tracking Glasses and Deep Learning

Konstantinos Roumpas and Michalis Xenos

In 2026 4th Cognitive Models and Artificial Intelligence Conference (AICCONF), Apr 2026

Abs DOI

Physics-Informed Neural Networks (PINNs) have gathered a lot of attention in the past years, as they have shown promising results in Physics, Chemistry, and other science-related projects. The application of such networks has also been explored in Human-Computer Interaction (HCI), and there is a lot of interest around eye-tracking (ET), as the partial differential equations (PDEs) that describe their movement are simplistic and can be easily applied in a PINN. Past research showed that with wearable ET devices, the PDEs that were described showed better results; however, there was no further study on them. Also, the potential of exploring different types of deep learning neural networks was discussed, although it was not elaborated further. In this work, we aim to apply Graph Neural Networks, Temporal Convolutional Networks, and their PINN-hybrid counterparts to explore if higher accuracy can be achieved in predicting a user’s gaze. The results showcased that the pure PINN is the best solution to tackling this problem, with the other neural networks falling shortly behind it. This is a significant finding for HCI research, as it showcases the most effective method for predicting a user’s gaze using ET data and deep learning, while also providing insight into the mechanics of eye movement itself.

2025

A comparative study of physics-informed and conventional neural networks for predicting on-screen gaze points from eye-tracking data

Konstantinos Roumpas, Eleftheria-Lito Michanetzi, Dimosthenis Minas, and 2 more authors

Expert Systems with Applications, Jul 2025

Abs DOI

Physics-Informed Neural Networks (PINNs) take advantage of physical constraints to improve the predictive performance of neural networks. They have demonstrated remarkable success in physics, chemistry, biology, and medicine, sparking interest in their potential applications in other fields, such as Human-Computer Interaction (HCI). Despite their versatility, PINNs are underutilized in certain disciplines due to the challenge of defining Partial Differential Equations (PDEs) that adequately constrain specific problems. This study investigates the application of PINNs to predict fixations captured by an eye-tracking device, employing equations that model head and eye movements. The data for this study were collected during an experiment involving an adaptive user interface for pilots, encompassing a variety of fixation points and rapid glances. The results establish PINNs as a superior method, achieving a mean absolute error (MAE) of 0.61 for horizontal predictions (X) and 0.35 for vertical predictions (Y), significantly outperforming the conventional neural network (NN). Additionally, the PINN demonstrated stronger predictive accuracy for vertical gaze dynamics, as evidenced by an RY2 of 0.91 compared to 0.85 for the NN. These findings underscore the robustness of the PINN approach in modeling gaze points with enhanced precision and reliability. While conventional neural networks perform adequately, the added benefit of physical constraints in PINNs encourages further exploration of their application in non-natural science domains. These promising results point toward the potential of PINNs in creating more adaptive and context-aware systems for various HCI applications.
Distinguishing Voluntary and Involuntary Blinks Through Eye Openness Metrics

Konstantinos Roumpas, Dimosthenis Minas, Alberto Calvo-Cordoba, and 1 more author

In Proceedings of the 3rd International Conference of the ACM Greek SIGCHI Chapter, Sep 2025

Abs DOI

Understanding and characterizing blinks can contribute to developing applications in human-computer interaction, fatigue monitoring, and medical diagnosis. This study involved the collection and analysis of eye-tracking data from 44 subjects to monitor voluntary and involuntary blinks. Using eye-openness metrics, our work builds on previous research that uses the same metric to monitor blinks. Blink duration thresholds were obtained from past studies involving eye images and were categorized as voluntary, involuntary, or invalid. In addition, a neural network model was developed and trained to classify blinks as voluntary or involuntary. The model achieved 92% accuracy on the test set, demonstrating the effectiveness of using eye-openness values for this classification task. This open-access dataset and model can provide a solid foundation for further research and applications in automated blink detection and analysis, and the potential creation of interactive systems that take voluntary blinks as input.