Gaze, physics and cognitive insight

An interactive version of our poster for the ACM GreekCHI-CYCHI Symposium 2026.

Poster · ACM GreekCHI-CYCHI Symposium 2026

Applying Physics-Informed Neural Networks and Eye-Tracking Data to Gain Cognitive Insights in Human-Centred Applications

Konstantinos Roumpas, Michalis Xenos · Software Quality & HCI Laboratory, Dept. of Computer Engineering & Informatics, University of Patras

Combining the equations governing the dynamics of an eye with eye-tracking data, we created an algorithm that essentially replicates the function of eye-tracking software. Along the way we compared different networks on the problem, and looked at what the model's least accurate moments have in common.

Move your pointer, or tap: the eye turns to follow it.

Objectives

01Model gaze with physics, not just data.
02Turn gaze into cognitive insight.
03Keep it responsible and sustainable.

How the model learns

A physics-informed neural network (PINN) is a network whose loss combines a standard data term with a physics term, derived from the equations governing the system. Pick a step to see what it does.

$$L_{\text{total}} = L_{\text{data}} + \lambda\, L_{\text{physics}}$$

The physics weight λ = 1.4 was tuned.

The guiding equation

The physics term encodes the eye's rotational dynamics on Donders' surface. Tap a symbol to see what it stands for.

$$H = \frac{\delta}{2}\,\Gamma_1^{T}\Gamma_1 + p^{T}F \qquad \Gamma = \frac{d}{dt}\frac{\partial \mathcal{L}}{\partial \dot{X}} - \frac{\partial \mathcal{L}}{\partial X}$$

Does it work?

The pure PINN achieves the lowest error on both axes, beating the feed-forward, GNN and TCN baselines and every PINN hybrid.

0.735
PINN mean absolute error, horizontal (X)
0.716
PINN mean absolute error, vertical (Y)
−64%
error on X compared with the feed-forward network

Mean absolute error by model

PINNother models

Lower is better. All differences vs. the PINN are significant (Wilcoxon, p < 0.01).

Predicted vs. actual gaze points

Six scatter panels of predicted against actual gaze points, one per model, coloured by Euclidean error

Colour shows the Euclidean error of each prediction, from blue (low) to red (high).

What's with the deviations?

We trained a classifier to predict when the physics model misses, using standard ocular features.

An AUC of 0.768 is well above chance, which establishes that the ocular features carry real joint information about when the physics model fails. It is also far from perfect, and it should be. The value of the residual is precisely that it captures something the standard features do not fully explain.
  • Blink countlargest positive weight
  • Saccade velocitypositive
  • Timesmall, positive
  • Pupilsmall, negative
  • Fixation durationlargest negative weight

Conclusions

Physics-informed constraints give the most accurate gaze model we tested.

Adding graph or temporal layers to the PINN increased error variance, which suggests the physics prior alone captures gaze dynamics most cleanly.

The approach is interpretable and data-efficient.

The same signals are a promising substrate for cognitive insight: attention, fatigue, comprehension.

This poster builds on our earlier work (Roumpas et al., 2025; Roumpas & Xenos, 2026). For the full research line, see cognitive modelling.

References

2026

  1. Modelling Eye Mechanics Using Eye-tracking Glasses and Deep Learning
    Konstantinos Roumpas and Michalis Xenos
    In 2026 4th Cognitive Models and Artificial Intelligence Conference (AICCONF), Apr 2026

2025

  1. A comparative study of physics-informed and conventional neural networks for predicting on-screen gaze points from eye-tracking data
    Konstantinos Roumpas, Eleftheria-Lito Michanetzi, Dimosthenis Minas, and 2 more authors
    Expert Systems with Applications, Jul 2025