Project Code Overview

The Gaze Engine is a software library designed to estimate and visualize a person's gaze direction. It processes video frames to detect faces, analyzes facial landmarks to determine head pose and eye direction, and combines these to calculate a final gaze vector. The engine relies heavily on OpenCV and various filtering techniques for smooth and stable results.

Key Components

Core Classes

• GazeEngine: The main engine class that orchestrates the entire pipeline, from image processing to applying visual features.
• EngineBase: A foundational class that handles common image processing tasks, resource initialization, and visualization.
• gaze::GazeEstimation: A key wrapper class that combines the outputs of the head and eye gaze estimators to produce a comprehensive gaze direction.
• core::Detection: A standard structure used to store details of a detected face, including its bounding box and facial landmarks.
• core::NeuralBase: The base class for neural network models used within the system, such as the face mesh detector.

Gaze Estimation Pipeline

gaze::HeadGazeEstimation: This class is responsible for calculating the head's 3D orientation (pose) from facial landmarks. It determines the direction the head is pointing, which serves as a baseline for the overall gaze.

gaze::EyeGazeEstimation: This class specifically analyzes the eye regions to estimate the direction of the pupils relative to the head. It also calculates the openness of each eye.

core::filter::PoseKalmanFilter: To ensure smooth and stable tracking of the head's position and rotation, a Kalman Filter is implemented. This filter reduces jitter from raw landmark data, providing a more realistic estimation of the head's pose over time.

Face Detection and Landmark Analysis

• core::face::FaceDetectorYunet: Utilizes OpenCV's FaceDetectorYN model to perform the initial, robust detection of faces within the video frame.
• core::face::FaceMesh: After a face is detected, this class processes it to generate a detailed mesh of facial landmarks, which are essential inputs for both head and eye gaze estimation.
• core::filter::LandmarkFlowFilter: This filter tracks landmarks across consecutive frames, providing temporal stability to the landmark positions.

Function Highlights

• Engine Initialization: GazeEngine::init(resourcePath) initializes the main engine and loads all necessary models and configurations.
• Image Processing: GazeEngine::process(input, output) executes the full gaze estimation pipeline on an input image and renders the results on the output image.
• Gaze Calculation: gaze::GazeEstimation::process(detection, drawingFrame) serves as the central function to process a detected face, invoking both head and eye estimators, and drawing the resulting gaze vectors.
• Head Pose Processing: gaze::HeadGazeEstimation::process(detection, drawingFrame) calculates and visualizes the head's pose.
• Eye Gaze Processing: gaze::EyeGazeEstimation::process(...) processes eye landmarks to determine gaze direction and eye openness.
• Pose Filtering: core::filter::PoseKalmanFilter::update(...) updates the Kalman filter with new raw pose data to return a smoothed and predicted pose.

File and Dependency Structure

The project files are organized in the C:/Projects/Engine/AntalEngine/Engine directory. Key header files include:

• GazeEngine.h: Defines the main GazeEngine class.
• GazeEstimation.h: Contains the wrapper class for combining gaze estimation methods.
• HeadGazeEstimation.h: Handles the calculation of head pose.
• EyeGazeEstimation.h: Handles the calculation of eye gaze and openness.
• PoseKalmanFilter.h: Defines the Kalman Filter used for smoothing pose data.
• FaceDetectorYunet.h: Provides the core face detection functionality.
• FaceMesh.h: Provides the facial landmark detection functionality.