diff --git a/README.md b/README.md index 108d9dd..75ec82c 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,53 @@ -OpenGL demo -=========== +# EyeLink Latency Characterization -To install compiler and library dependencies on Debian/Ubuntu: +A set of simple hardware and software used to characterize end-to-end latency of +an EyeLink 1000 eye tracker and the Xubuntu 18.04 graphics stack with +compositing disabled. -``` -$ sudo apt install build-essential autotools-dev autoconf automake libgl-dev libglfw3-dev libglew-dev libglu-dev -``` +## Getting Started + +### Prerequisites + +* **EyeLink Developers Kit for Linux** + Steps to install: + + 1. Add signing key + ``` + $ wget -O - "http://download.sr-support.com/software/dists/SRResearch/SRResearch_key" | sudo apt-key add - + ``` + 2. Add apt repository + ``` + $ sudo add-apt-repository "deb http://download.sr-support.com/software SRResearch main" + $ sudo apt-get update + ``` + 3. Install latest release of EyeLink Developers Kit for Linux + ``` + $ sudo apt-get install eyelink-display-software + ``` + Alternatively, a tar of DEBs is available [at this link][debs]. + +* **Network Configuration on Host PC** + The EyeLink PC and the Host PC communicate via a direct Ethernet connection + between the two computers. In order for third party programming + packages/languages to use this connection, you must use a static IP address for + the Ethernet port that is used to connect to the Host PC. This static IP should + be: + + ``` + IP Address: 100.1.1.2 + Subnet Mask: 255.255.255.0 + ``` +* **Host Software Dependencies** + To install compiler and library dependencies on Debian/Ubuntu: + + ``` + $ sudo apt install build-essential autotools-dev autoconf automake libgl-dev libglfw3-dev libglew-dev libglu-dev + ``` +* **Arduino IDE** + To install and set up the Arduino IDE, follow [this guide][arduino-guide], and + pay special attention to setting serial port permissions. + +### Host Software To compile: @@ -23,4 +65,130 @@ To use (from inside `build` directory): $ ./src/frontend/example ``` -Main source code to read: [src/frontend/example.cc](https://github.com/keithw/gldemo/blob/master/src/frontend/example.cc) +This will log the timing results to a file (results.csv) that can be analysed +afterwords to visualize the latency distributions. The CSV file format is + +``` +e2e (us),eyelink (us),drawing (us) +6852,1820,635 +9528,1616,595 +7180,1656,618 +5504,1194,632 +... +``` + +Main source code to read: [src/frontend/example.cc](src/frontend/example.cc). + +### Artificial Saccade Generator Software + +To setup the Arduino for use as the artificial saccade generator, use Arduino +IDE on the Host computer to program the Arduino with the script found in +[scripts/arduino.ino](scripts/arduino.ino). + +## Reference + +The following sections are information-oriented. They detail the specifics of +the setup. + +### End-to-end Latency Measurement + +**Bill of Materials** + +* [1x] Artificial Saccade Generator (ASG), described below. +* [1x] Computer running Xubuntu 18.04 with compositing DISABLED. +* [1x] Dell P2815Q Display running 1920x1080 @ 240Hz using Zisworks x28 R2 kit. + +After the artificial saccade, we use a host computer to continuously monitor for +gaze position changes. Once a change is detected, we use OpenGL to quickly +switch a portion of the display white. A photodiode is mounted to the area of +the display that will change. Once the photodiode triggers on the display +change, we take the time difference between the display change event and +toggling the IR LEDs as a measure of the end-to-end latency of the system. The +sequence diagram below gives an overview of the process. + +``` + ┌────┐ ┌───┐ ┌───────┐ + │Host│ │ASG│ │Display│ + └─┬──┘ └─┬─┘ └───┬───┘ + │ Switch Eye Command │ │ + │ ────────────────────────────────> │ + │ │ │ + │ ╔═══════════════╧════════════════╗ │ + │ ║toggle IR LED, t_start = now() ░║ │ + │ ╚═══════════════╤════════════════╝ │ + │────┐ │ + │ │ poll EyeLink for new samples │ + │<───┘ │ + │ │ │ + │ │────┐ + │ │ │ sample = photodiode level + │ │<───┘ + │ │ │ +╔════════════╧═══════════════╗ │ │ +║new EyeLink Sample > DELTA ░║ │ │ +╚════════════╤═══════════════╝ │ │ + │ draw white box │ + │ ───────────────────────────────────────────────────────────────> + │ │ │ + │ ╔══════════════════╧════════════════════╗ │ + │ ║sample > trigger level, t_end = now() ░║ │ + │ ╚══════════════════╤════════════════════╝ │ + │ t_end - t_start │ │ + │ <──────────────────────────────── │ + ┌─┴──┐ ┌─┴─┐ ┌───┴───┐ + │Host│ │ASG│ │Display│ + └────┘ └───┘ └───────┘ +``` + +A table clamp is used to secure the ASG to the table top so that it is easy to +tune the resistance of the potentiometers, and pupil sizes (i.e., changing +EyeLink settings, or printing different sized black dots on paper), such that +the EyeLink detects the artificial pupil reliably. This is shown below. We also +found that we needed to block some of the EyeLink's IR LED array so that the +reflection off of the white paper did not hide the IR LEDs from the ASG. + +![setup][setup] + +### Artificial Saccade Generator + +To characterize the end-to-end latency, we use an artificial saccade generator, +which works by tricking the eye tracker to detect an abrupt change in the gaze +position of an artificial eye which does not move (a technique described in +[[1]]). The key insight is that most video-based eye trackers compute gaze +position by using the position of the pupil (emulated by a black dot laser +printed on white copy paper) as well as the position of a corneal reflection +(emulated by an infrared LED). By using two IR LEDs, we can programmatically +trigger an artificial saccade by switching one IR LED off, while simultaneously +switching the other IR LED on. + +We implement this using an Arduino Uno with a custom circuit soldered to a +prototyping shield. + +**Bill of Materials** + +* [1x] F12N10L N-mosfet +* [1x] 1RF9530 P-mosfet +* [2x] Trim potentiometers +* [1x] 2.2 M-ohm resistor +* [1x] 22pF capacitor +* [2x] Gikfun 5mm 940nm IR LEDs +* [1x] FDS100 Photodiode +* [1x] Mount + standoffs +* [1x] Arduino Uno +* [1x] Adafruit Proto Shield for Arduino +* [1x] Black pupil laser printed on white paper + +The circuit that is soldered to the Arduino Proto Shield is shown below. + +![schematic][logo] + +The proto shield is then mounted to the Arduino, and the two are then mounted to +an acrylic piece of plastic, with two holes cut to allow the IR LEDs to shine +through from behind the artificial eye. The laser cutter SVG can be found in +[docs/mount.svg](docs/mount.svg). + +[1]: https://www.ncbi.nlm.nih.gov/pubmed/24771998 +[arduino-guide]: https://www.arduino.cc/en/guide/linux +[debs]: http://download.sr-support.com/linuxDisplaySoftwareRelease/eyelink-display-software_1.11_x64_debs.tar.gz +[logo]: docs/circuit.png +[setup]: docs/setup.jpg diff --git a/configure.ac b/configure.ac index 9f7c93a..14c4e65 100644 --- a/configure.ac +++ b/configure.ac @@ -28,6 +28,15 @@ PKG_CHECK_MODULES([GL], [gl]) PKG_CHECK_MODULES([GLU], [glu]) PKG_CHECK_MODULES([GLFW3], [glfw3]) PKG_CHECK_MODULES([GLEW], [glew]) +PKG_CHECK_MODULES([OPENCV], [opencv]) +PKG_CHECK_MODULES([PANGOCAIRO], [pangocairo]) +PKG_CHECK_MODULES([SDL], [sdl]) +PKG_CHECK_MODULES([SDL_GFX], [SDL_gfx]) +PKG_CHECK_MODULES([SDL_TTF], [SDL_ttf]) +PKG_CHECK_MODULES([SDL_IMAGE], [SDL_image]) +PKG_CHECK_MODULES([SDL_MIXER], [SDL_mixer]) +PKG_CHECK_MODULES([MSGPACK], [msgpack]) +PKG_CHECK_MODULES([LIBZMQ], [libzmq]) # Checks for header files. AC_LANG_PUSH(C++) diff --git a/docs/circuit.png b/docs/circuit.png new file mode 100644 index 0000000..e8236fe Binary files /dev/null and b/docs/circuit.png differ diff --git a/docs/mount.svg b/docs/mount.svg new file mode 100644 index 0000000..8bef1f1 --- /dev/null +++ b/docs/mount.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/setup.jpg b/docs/setup.jpg new file mode 100644 index 0000000..1588b76 Binary files /dev/null and b/docs/setup.jpg differ diff --git a/scripts/analysis/analysis.py b/scripts/analysis/analysis.py new file mode 100755 index 0000000..86d1010 --- /dev/null +++ b/scripts/analysis/analysis.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python +import argparse +import logging +import sys +from subprocess import DEVNULL, run + +import matplotlib +import matplotlib.pyplot as plt +import pandas as pd +import seaborn as sns +import numpy as np +from matplotlib.backends.backend_pdf import PdfPages + +matplotlib.rcParams["text.usetex"] = True + +sns.set(style="whitegrid") +sns.set_context("paper", font_scale=1.5, rc={"lines.linewidth": 2.25}) + + +# Configure logging +logging.basicConfig( + stream=sys.stdout, + format="[%(asctime)s][%(levelname)s] %(name)s:%(lineno)s - %(message)s", + level=logging.INFO, +) +logger = logging.getLogger(__name__) + +FIGSIZE = (7, 4) + + +def _plot(infile): + """Plotting logic.""" + fig, ax = plt.subplots(figsize=FIGSIZE) + + # type,b,f1,precision,recall + data = pd.read_csv(infile, skipinitialspace=True) + + # Plot PDF + plot = sns.distplot( + data["e2e (us)"] / 1000.0, + kde=False, + bins=25, + ax=ax + ) + + sns.despine(bottom=True, left=True) + plot.set(xlabel="End-to-end Latency (ms)") + plot.set(ylabel="Histogram") + outfile = "histogram.pdf" + pp = PdfPages(outfile) + pp.savefig(plot.get_figure().tight_layout()) + pp.close() + # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True) + logger.info(f"Plot saved to {outfile}") + + # Plot Boxplot + fig, ax = plt.subplots(figsize=FIGSIZE) + plot = sns.boxplot( + data = data, + orient = "h", + ax=ax + ) + + sns.despine(bottom=True, top=True) + plot.set(ylabel="Delay Type") + plot.set(xlabel="Time (us)") + outfile = "boxplot.pdf" + pp = PdfPages(outfile) + pp.savefig(plot.get_figure().tight_layout()) + pp.close() + # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True) + logger.info(f"Plot saved to {outfile}") + + # Plot CDF + fig, ax = plt.subplots(figsize=FIGSIZE) + plot = sns.distplot( + data["e2e (us)"] / 1000.0, + hist_kws={"cumulative": True, "rwidth": 0.85}, + norm_hist=True, + # bins = 45, + kde=False + ) + + # handles, labels = ax.get_legend_handles_labels() + # ax.legend(handles=handles[1:], labels=labels[1:]) + ax.set_ylim([0, 1]) + # ax.set_xlim([0.5, 1]) + + sns.despine(bottom=True, left=True) + plot.set(xlabel="End-to-end Latency (ms)") + plot.set(ylabel="Cumulative Probability") + outfile = "cdf.pdf" + pp = PdfPages(outfile) + pp.savefig(plot.get_figure().tight_layout()) + pp.close() + # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True) + logger.info(f"Plot saved to {outfile}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--data", type=str, default="../../results.csv", help="CSV file of latency data" + ) + parser.add_argument( + "-v", + "--verbose", + dest="verbose", + action="store_true", + help="Output INFO level logging.", + ) + args = parser.parse_args() + + if args.verbose: + ch = logging.StreamHandler() + logger.setLevel(logging.INFO) + ch.setLevel(logging.INFO) + formatter = logging.Formatter("[%(levelname)s] %(name)s - %(message)s") + ch.setFormatter(formatter) + logger.addHandler(ch) + + _plot(args.data) diff --git a/scripts/analysis/requirements.txt b/scripts/analysis/requirements.txt new file mode 100644 index 0000000..19bdfa0 --- /dev/null +++ b/scripts/analysis/requirements.txt @@ -0,0 +1,3 @@ +matplotlib +pandas +seaborn diff --git a/scripts/arduino.ino b/scripts/arduino.ino new file mode 100644 index 0000000..54fffc4 --- /dev/null +++ b/scripts/arduino.ino @@ -0,0 +1,56 @@ +// These pin numbers depend on which pins you use for your circuit +int sensorPin = A0; +int ledPin = 10; + +void setup() +{ + // Use a 115200 baud rate + Serial.begin(115200); + pinMode(ledPin, OUTPUT); + pinMode(sensorPin, INPUT); +} + +void loop() +{ + static uint32_t ts1 = 0; + static uint32_t ts2 = 0; + static int state = 0; + static int sensorValue = 0; + int cmd = 0; + + switch (state) { + case 0: + digitalWrite(ledPin, HIGH); + + // poll for new command + if (Serial.available() > 0) { + cmd = Serial.read(); + + // command from the host PC is hard-coded as the character 'g' + if (cmd == 'g') { + // Start timer + ts1 = micros(); + + // Move to next state + state = 1; + } + } + break; + + case 1: + digitalWrite(ledPin, LOW); + sensorValue = analogRead(sensorPin); + + // Rising edge trigger condition + if (sensorValue > 512) { + // Log time difference and send back to host PC + ts2 = micros(); + Serial.println(ts2 - ts1); + + // Reset state + state = 0; + sensorValue = 0; + } + break; + } +} diff --git a/src/files/frame92.png b/src/files/frame92.png new file mode 100644 index 0000000..e60a752 Binary files /dev/null and b/src/files/frame92.png differ diff --git a/src/files/frame92_resized.png b/src/files/frame92_resized.png new file mode 100644 index 0000000..64dace1 Binary files /dev/null and b/src/files/frame92_resized.png differ diff --git a/src/files/testim.png b/src/files/testim.png new file mode 100644 index 0000000..c16abb2 Binary files /dev/null and b/src/files/testim.png differ diff --git a/src/frontend/Makefile.am b/src/frontend/Makefile.am index 8ca7140..b2098b3 100644 --- a/src/frontend/Makefile.am +++ b/src/frontend/Makefile.am @@ -1,7 +1,22 @@ -AM_CPPFLAGS = $(CXX17_FLAGS) $(SSL_CFLAGS) -I$(srcdir)/../util +AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLEW_CFLAGS) $(GLFW3_CFLAGS) $(PANGOCAIRO_CFLAGS) $(SDL_CFLAGS) $(SDL_GFX_CFLAGS) $(SDL_IMAGE_CFLAGS) $(SDL_TTF_CFLAGS) $(SDL_MIXER_CFLAGS) $(LIBZMQ_CFLAGS) $(MSGPACK_CFLAGS) $(OPENCV_CFLAGS) -I$(srcdir)/../util AM_CXXFLAGS = $(PICKY_CXXFLAGS) -bin_PROGRAMS = example +bin_PROGRAMS = example drawtext user_test pupil_labs gazedraw equitest example_SOURCES = example.cc -example_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) +example_LDADD = -L/usr/lib -leyelink_core_graphics -leyelink_core -lsdl_util -lrt -leyelink_core_graphics -lpthread ../util/libgldemoutil.a $(SDL_LIBS) $(SDL_GFX_LIBS) $(SDL_IMAGE_LIBS) $(SDL_TTF_LIBS) $(SDL_MIXER_LIBS) $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) + +drawtext_SOURCES = drawtext.cc +drawtext_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) + +user_test_SOURCES = user_test.cc +user_test_LDADD = -L/usr/lib -leyelink_core_graphics -leyelink_core -lsdl_util -lrt -leyelink_core_graphics -lpthread ../util/libgldemoutil.a $(SDL_LIBS) $(SDL_GFX_LIBS) $(SDL_IMAGE_LIBS) $(SDL_TTF_LIBS) $(SDL_MIXER_LIBS) $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) + +pupil_labs_SOURCES = pupil_labs.cc +pupil_labs_LDADD = $(LIBZMQ_LIBS) $(MSGPACK_LIBS) + +gazedraw_SOURCES = gazedraw.cc +gazedraw_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) $(LIBZMQ_LIBS) $(MSGPACK_LIBS) + +equitest_SOURCES = equitest.cc +equitest_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) $(OPENCV_LIBS) \ No newline at end of file diff --git a/src/frontend/drawtext.cc b/src/frontend/drawtext.cc new file mode 100644 index 0000000..1bba2d8 --- /dev/null +++ b/src/frontend/drawtext.cc @@ -0,0 +1,74 @@ +#include +#include +#include +#include +#include + +#include + +#include "cairo_objects.hh" +#include "display.hh" + +using namespace std; +using namespace std::chrono; + +void program_body() +{ + VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution + + Cairo cairo { 1920, 1080 }; + Pango pango { cairo }; + + /* open the PNG */ + PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/testim.png" }; + + /* draw gray over everything */ + cairo_new_path( cairo ); + cairo_identity_matrix( cairo ); + cairo_rectangle( cairo, 500, 500, 100, 100 ); + cairo_set_source_rgba( cairo, 0, 0.9, 0, 0.5 ); + cairo_fill( cairo ); + + /* draw the PNG */ + cairo_identity_matrix( cairo ); + cairo_scale( cairo, 0.8, 0.8 ); + double center_x = 960, center_y = 540; + cairo_device_to_user( cairo, ¢er_x, ¢er_y ); + cairo_translate( cairo, center_x, center_y ); + cairo_set_source_surface( cairo, png_image, 0, 0 ); + cairo_paint( cairo ); + + /* draw some text */ + Pango::Font myfont { "Times New Roman, 80" }; + Pango::Text mystring { cairo, pango, myfont, "Hello, world, Brooke, and Luke." }; + mystring.draw_centered_at( cairo, 960, 540 ); + cairo_set_source_rgba( cairo, 1, 1, 1, 1 ); + cairo_fill( cairo ); + + /* finish and copy to YUV raster */ + cairo.flush(); + + unsigned int stride = cairo.stride(); + Raster420 yuv_raster { 1920, 1080 }; + for ( unsigned int y = 0; y < 1080; y++ ) { + for ( unsigned int x = 0; x < 1920; x++ ) { + yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )]; + } + } + + Texture420 texture { yuv_raster }; + display.draw( texture ); + pause(); +} + +int main() +{ + try { + program_body(); + } catch ( const exception& e ) { + cerr << "Exception: " << e.what() << "\n"; + return EXIT_FAILURE; + } + + return EXIT_SUCCESS; +} diff --git a/src/frontend/equitest.cc b/src/frontend/equitest.cc new file mode 100644 index 0000000..2b75317 --- /dev/null +++ b/src/frontend/equitest.cc @@ -0,0 +1,212 @@ + +#include +#include +#include +#include +#include + +#include + +#include "cairo_objects.hh" +#include "display.hh" + +#include "opencv2/core/core.hpp" +#include "opencv2/highgui/highgui.hpp" +#include "opencv2/imgproc/imgproc.hpp" + +using namespace std; +using namespace std::chrono; + +#define SCREEN_RES_X 1920 +#define SCREEN_RES_Y 1080 +#define pi 3.14159265 + +void writePNGRaster(Raster420 & yuv_raster) { + Cairo cairo { SCREEN_RES_X, SCREEN_RES_Y }; + + /* open the PNG */ + PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/frame92_resized.png" }; + + /* draw the PNG */ + cairo_identity_matrix( cairo ); + //cairo_scale( cairo, 0.234375, 0.263671875 ); + //cairo_scale( cairo, 0.5, 0.52734375 ); + cairo_scale(cairo, 1.0, 1.0); + double center_x = 0, center_y = 0; + cairo_device_to_user( cairo, ¢er_x, ¢er_y ); + cairo_translate( cairo, center_x, center_y ); + cairo_set_source_surface( cairo, png_image, 0, 0 ); + cairo_paint( cairo ); + + /* finish and copy to YUV raster */ + cairo.flush(); + + unsigned int stride = cairo.stride(); + for ( unsigned int y = 0; y < SCREEN_RES_Y; y++ ) { + for ( unsigned int x = 0; x < SCREEN_RES_X; x++ ) { + float red = cairo.pixels()[y * stride + 2 + ( x * 4 )] / 255.0; + float green = cairo.pixels()[y * stride + 1 + ( x * 4 )] / 255.0; + float blue = cairo.pixels()[y * stride + 0 + ( x * 4 )] / 255.0; + + const float Ey = 0.7154 * green + 0.0721 * blue + 0.2125 * red; + const float Epb = -0.386 * green + 0.5000 * blue - 0.115 * red; + const float Epr = -0.454 * green - 0.046 * blue + 0.500 * red; + + const uint8_t Y = (219 * Ey) + 16; + const uint8_t Cb = (224 * Epb) + 128; + const uint8_t Cr = (224 * Epr) + 128; + + yuv_raster.Y.at( x, y ) = Y; + if ( (x%2) == 0 and (y%2) == 0 ) { + yuv_raster.Cb.at( x / 2, y / 2 ) = Cb; + yuv_raster.Cr.at( x / 2, y / 2 ) = Cr; + } + } + } +} + +cv::Mat eul2rotm(double rotx, double roty, double rotz) +{ + cv::Mat R_x = (cv::Mat_(3, 3) << 1, 0, 0, + 0, cos(rotx), -sin(rotx), + 0, sin(rotx), cos(rotx)); + + cv::Mat R_y = (cv::Mat_(3, 3) << cos(roty), 0, sin(roty), + 0, 1, 0, + -sin(roty), 0, cos(roty)); + + cv::Mat R_z = (cv::Mat_(3, 3) << cos(rotz), -sin(rotz), 0, + sin(rotz), cos(rotz), 0, + 0, 0, 1); + cv::Mat K = (cv::Mat_(3, 3) << 672, 0, SCREEN_RES_X / 2, + 0, 672, SCREEN_RES_Y / 2, + 0, 0, 1); + + cv::Mat R = R_z * R_y * R_x * K.inv(); + + return R; +} + +cv::Mat xyz_norm_map_x; +cv::Mat xyz_norm_map_y; + +std::pair reprojection(int x_img, int y_img, cv::Mat& img_src) +{ + cv::Mat xyz = (cv::Mat_(3, 1) << (double)x_img, (double)y_img, 1); + cv::Mat xyz_norm = xyz / norm(xyz); + + //xyz_norm_map_x.at(x_img, y_img) = xyz_norm.at(0,0); + + //std::cout << "[ " << xyz_norm.at(0, 0) << ", " <(0, 1) << ", " <(0, 2) << " ]" << std::endl; + + cv::Mat Rot = eul2rotm(0.5, 0, 0); + + //std::cout << Rot << std::endl; + + cv::Mat ray3d = Rot * xyz_norm; + + //get 3d spherical coordinates + double xp = ray3d.at(0, 0); + double yp = ray3d.at(0, 1); + double zp = ray3d.at(0, 2); + //inverse formula for spherical projection, reference Szeliski book "Computer Vision: Algorithms and Applications" p439. + double theta = atan2(yp, sqrt(xp * xp + zp * zp)); + double phi = atan2(xp, zp); + + //get 2D point on equirectangular map + double x_sphere = (((phi * img_src.cols) / pi + img_src.cols) / 2); + double y_sphere = (theta + pi / 2) * img_src.rows / pi; + + return std::make_pair(x_sphere, y_sphere); +} + +void rectPortionRaster(Raster420 & yuv_raster, cv::Mat& img_src) { + + for ( unsigned int y = 0; y < SCREEN_RES_Y; y++ ) { + for ( unsigned int x = 0; x < SCREEN_RES_X; x++ ) { + //determine corresponding position in the equirectangular panorama + std::pair current_pos = reprojection(x, y, img_src); + + //extract the x and y value of the position in the equirect. panorama + double current_x = current_pos.first; + double current_y = current_pos.second; + + // if the current position exceeeds the panorama image limit -- leave pixel black and skip to next iteration + if (current_x < 0 || current_y < 0 ) + { + continue; + } + + cv::Vec3b bgr = img_src.at(current_y, current_x); + + float blue = bgr.val[0]/255.0; + float green = bgr.val[1]/255.0; + float red = bgr.val[2]/255.0; + + const float Ey = 0.7154 * green + 0.0721 * blue + 0.2125 * red; + const float Epb = -0.386 * green + 0.5000 * blue - 0.115 * red; + const float Epr = -0.454 * green - 0.046 * blue + 0.500 * red; + + const uint8_t Y = (219 * Ey) + 16; + const uint8_t Cb = (224 * Epb) + 128; + const uint8_t Cr = (224 * Epr) + 128; + + yuv_raster.Y.at( x, y ) = Y; + if ( (x%2) == 0 and (y%2) == 0 ) { + yuv_raster.Cb.at( x / 2, y / 2 ) = Cb; + yuv_raster.Cr.at( x / 2, y / 2 ) = Cr; + } + } + } + return; +} + +void onlineMethod() { + cv::Mat img_src = cv::imread("/home/brooke/repos/eyelink-latency/src/files/frame92.png", cv::IMREAD_COLOR); + VideoDisplay display { SCREEN_RES_X, SCREEN_RES_Y, false }; // fullscreen window @ 1920x1080 luma resolution + Raster420 yuv_raster { SCREEN_RES_X , SCREEN_RES_Y }; + rectPortionRaster(yuv_raster, img_src); + Texture420 texture { yuv_raster }; + + //std::cout << img_src.cols << std::endl; + + while (true) { + display.draw( texture ); + } +} + +void withShader() { + + VideoDisplay display { SCREEN_RES_X, SCREEN_RES_Y, false }; // fullscreen window @ 1920x1080 luma resolution + Raster420 yuv_raster { SCREEN_RES_X, SCREEN_RES_Y }; + writePNGRaster(yuv_raster); + Texture420 texture { yuv_raster }; + + float roll = 0; + float pitch = 0; + float yaw = 0; + + while (true) { + display.draw( texture ); + display.update_head_orientation( roll, pitch, yaw ); + roll += 0.001; + pitch += 0.003; + yaw += 0.001; + } + +} + + +int main() +{ + try { + withShader(); + //onlineMethod(); + + } catch ( const exception& e ) { + cerr << "Exception: " << e.what() << "\n"; + return EXIT_FAILURE; + } + + return EXIT_SUCCESS; +} diff --git a/src/frontend/example.cc b/src/frontend/example.cc index f2afd06..3956da1 100644 --- a/src/frontend/example.cc +++ b/src/frontend/example.cc @@ -1,63 +1,336 @@ +#include #include #include #include +#include #include +#include + +#include +#include +#include +#include + +#include +#include +#include #include "display.hh" +#define CURSOR_SIZE 5 /* radius of the white dot in px */ +#define NUM_TRIALS 1 + using namespace std; using namespace std::chrono; -void program_body() +/** + * End recording: adds 100 msec of data to catch final events + */ +static void end_trial( void ) { - VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution - display.window().hide_cursor( true ); - display.window().set_swap_interval( 1 ); // wait for vertical retrace before swapping buffer + pump_delay( 100 ); // provide a small amount of delay for last data + stop_recording(); + while ( getkey() ) { + }; +} + +int get_tracker_sw_version( char* verstr ) +{ + int ln = 0; + int st = 0; + ln = strlen( verstr ); + while ( ln > 0 && verstr[ln - 1] == ' ' ) + verstr[--ln] = 0; // trim + + // find the start of the version number + st = ln; + while ( st > 0 && verstr[st - 1] != ' ' ) + st--; + return atoi( &verstr[st] ); +} + +int initialize_eyelink() +{ + char verstr[50]; + int eyelink_ver = 0; + int tracker_software_ver = 0; + + // Set the address of the tracker, this is hard-coded by the Eyelink + set_eyelink_address( (char*)"100.1.1.1" ); + + // Initialize the EyeLink DLL and connect to the tracker + // * 0 opens a connection with the eye tracker + // * 1 will create a dummy connection for simulation + // * -1 initializes the DLL but does not open a connection + if ( open_eyelink_connection( 0 ) ) + return -1; - /* all white (235 = max luma in typical Y'CbCr colorspace) */ - Raster420 white { 1920, 1080 }; - memset( white.Y.mutable_pixels(), 235, white.Y.width() * white.Y.height() ); - Texture420 white_texture { white }; + set_offline_mode(); + flush_getkey_queue(); - /* left half white */ - Raster420 left_white { 1920, 1080 }; + // Now configure tracker for display resolution + eyecmd_printf( "screen_pixel_coords = %ld %ld %ld %ld", 0, 0, 1920, 1080 ); - for ( unsigned int y = 0; y < left_white.Y.height(); y++ ) { - for ( unsigned int x = 0; x < left_white.Y.width(); x++ ) { - const uint8_t color = ( x < left_white.Y.width() / 2 ) ? 235 : 16; - left_white.Y.at( x, y ) = color; + eyelink_ver = eyelink_get_tracker_version( verstr ); + if ( eyelink_ver == 3 ) + tracker_software_ver = get_tracker_sw_version( verstr ); + + // SET UP TRACKER CONFIGURATION + // set parser saccade thresholds (conservative settings) + if ( eyelink_ver >= 2 ) { + eyecmd_printf( "select_parser_configuration 0" ); // 0 = standard sensitivity + // turn off scenelink camera stuff + if ( eyelink_ver == 2 ) { + eyecmd_printf( "scene_camera_gazemap = NO" ); } + } else { + eyecmd_printf( "saccade_velocity_threshold = 35" ); + eyecmd_printf( "saccade_acceleration_threshold = 9500" ); + } + + // set link data (used for gaze cursor) + eyecmd_printf( "link_event_filter = LEFT,RIGHT,FIXATION,SACCADE,BLINK,BUTTON,INPUT" ); + eyecmd_printf( "link_sample_data = LEFT,RIGHT,GAZE,GAZERES,AREA,STATUS%s,INPUT", + ( tracker_software_ver >= 4 ) ? ",HTARGET" : "" ); + + // Make sure we're still alive + if ( !eyelink_is_connected() || break_pressed() ) { + return -1; + } + return 0; +} + +int exit_eyelink() +{ + close_expt_graphics(); // tell EXPTSPPT to release window + close_eyelink_connection(); // disconnect from tracker + return 0; +} + +void clear_full_screen_window( SDL_Surface* window, SDL_Color c ) +{ + SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) ); + SDL_Flip( window ); + SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) ); +} + +void do_calibration() +{ + // The colors of the target and background for calibration and drift correct + SDL_Color target_background_color = { 192, 192, 192, 0 }; + SDL_Color target_foreground_color = { 0, 0, 0, 0 }; + SDL_Surface* window = NULL; + + // register window with EXPTSPPT + if ( init_expt_graphics( NULL, NULL ) ) { + exit_eyelink(); + return; } - Texture420 left_white_texture { left_white }; + // Setup calibration type + eyecmd_printf( "calibration_type = HV9" ); - /* all black (16 = min luma in typical Y'CbCr colorspace) */ - Raster420 black { 1920, 1080 }; - memset( black.Y.mutable_pixels(), 16, black.Y.width() * black.Y.height() ); - Texture420 black_texture { black }; + window = SDL_GetVideoSurface(); - /* alternate black and white */ + // Size for calibration target and focal spot + unsigned int i = 1920 / 60; + unsigned int j = 1920 / 300; + if ( j < 2 ) + j = 2; + set_target_size( i, j ); // tell DLL the size of target features + + // tell EXPTSPPT the colors + set_calibration_colors( &target_foreground_color, &target_background_color ); + + clear_full_screen_window( window, target_background_color ); + + SDL_Flip( window ); + + do_tracker_setup(); + + // Wait for user to press escape before continuing + while ( !escape_pressed() ) { + }; + + close_expt_graphics(); // tell EXPTSPPT to release window +} + +int gc_window_trial() +{ + ofstream log; + log.open( "results.csv" ); unsigned int frame_count = 0; + int button; /* the button pressed (0 if timeout) */ + + // First, set up all the textures + VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution + display.window().hide_cursor( true ); + + // whether to wait for vertical retrace before swapping buffer + // * 0 for immediate updates + // * 1 for updates synchronized with the vertical retrace + // * -1 for adaptive vsync + display.window().set_swap_interval( 0 ); + + // Used to track gaze samples + ALLF_DATA evt; + float x_sample, y_sample; + + // Ensure Eyelink has enough time to switch modes + set_offline_mode(); + pump_delay( 50 ); + + // Start data streaming + // Note that we are ignoring the EDF file. + int error = start_recording( 0, 0, 1, 1 ); + if ( error != 0 ) { + log.close(); + return error; + } + + // wait for link sample data + if ( !eyelink_wait_for_block_start( 100, 1, 0 ) ) { + end_trial(); + cerr << "ERROR: No link samples received!\n"; + log.close(); + return TRIAL_ERROR; + } + + // determine which eye(s) are available + int eye_used = eyelink_eye_available(); + + // reset keys and buttons from tracker + eyelink_flush_keybuttons( 0 ); const auto start_time = steady_clock::now(); + // Poll for new samples until the diff between samples is large enough to signify LEDs switched while ( true ) { - display.draw( left_white_texture ); - frame_count++; - display.draw( white_texture ); - frame_count++; - display.draw( black_texture ); - frame_count++; + // Termination conditions + if ( ( error = check_recording() ) != 0 ) { + log.close(); + return error; + } + + if ( break_pressed() ) /* check for program termination or ALT-F4 or CTRL-C keys */ + { + end_trial(); /* local function to stop recording */ + log.close(); + return ABORT_EXPT; /* return this code to terminate experiment */ + } + + if ( escape_pressed() ) /* check for local ESC key to abort trial (useful in debugging) */ + { + end_trial(); /* local function to stop recording */ + log.close(); + return SKIP_TRIAL; /* return this code if trial terminated */ + } + + /* BUTTON RESPONSE TEST */ + /* Check for eye-tracker buttons pressed */ + /* This is the preferred way to get response data or end trials */ + button = eyelink_last_button_press( NULL ); + if ( button != 0 ) /* button number, or 0 if none pressed */ + { + end_trial(); /* local function to stop recording */ + break; /* exit trial loop */ + } + + // check for new sample update + if ( eyelink_newest_float_sample( NULL ) > 0 ) { + eyelink_newest_float_sample( &evt ); + + x_sample = evt.fs.gx[eye_used]; + y_sample = evt.fs.gy[eye_used]; - if ( frame_count % 480 == 0 ) { - const auto now = steady_clock::now(); + const auto t_sample = duration_cast( steady_clock::now() - start_time ).count(); + log << t_sample << ", " << x_sample << ", " << y_sample << endl; - const auto ms_elapsed = duration_cast( now - start_time ).count(); + // make sure pupil is present + if ( x_sample != MISSING_DATA && y_sample != MISSING_DATA && evt.fs.pa[eye_used] > 0 ) { + // TODO: Draw a dot where we are looking. This is currently very naive, + // we should be able to draw a dot faster. Right now, this is limiting + // our FPS to about 77fps. + + /* draw cursor location (235 = max luma in typical Y'CbCr colorspace) */ + Raster420 cursor { 1920, 1080 }; - cout << "Drew " << frame_count << " frames in " << ms_elapsed - << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n"; + for ( unsigned int y = 0; y < cursor.Y.height(); y++ ) { + for ( unsigned int x = 0; x < cursor.Y.width(); x++ ) { + // Check if point is within radius + bool within_circle = ( ( ( x_sample - x ) * ( x_sample - x ) ) + + ( ( y_sample - y ) * ( y_sample - y ) ) ) <= ( CURSOR_SIZE * CURSOR_SIZE ); + const uint8_t color = within_circle ? 235 : 16; + cursor.Y.at( x, y ) = color; + } + } + + Texture420 cursor_texture { cursor }; + + // Draw texture. Note this may introduce extra delay since this is + // computing a texture each update. + display.draw( cursor_texture ); + + frame_count++; + + if ( frame_count % 480 == 0 ) { + const auto now = steady_clock::now(); + const auto ms_elapsed = duration_cast( now - start_time ).count(); + cout << "Drew " << frame_count << " frames in " << ms_elapsed + << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n"; + } + } + } + } + end_trial(); + log.close(); + return check_record_exit(); +} + +int run_trials() +{ + + do_calibration(); + + for ( unsigned int trial = 0; trial < NUM_TRIALS; trial++ ) { + // abort if link is closed + if ( eyelink_is_connected() == 0 || break_pressed() ) { + return ABORT_EXPT; } + + int i = gc_window_trial(); + + // Report errors + switch ( i ) { + case ABORT_EXPT: // handle experiment abort or disconnect + cout << "EXPERIMENT ABORTED\n"; + return ABORT_EXPT; + case REPEAT_TRIAL: // trial restart requested + cout << "TRIAL REPEATED\n"; + trial--; + break; + case SKIP_TRIAL: // skip trial + cout << "TRIAL ABORTED\n"; + break; + case TRIAL_OK: // successful trial + cout << "TRIAL OK\n"; + break; + default: // other error code + cout << "TRIAL ERROR\n"; + break; + } + } + + return 0; +} + +void program_body() +{ + if ( initialize_eyelink() < 0 ) { + cerr << "[Error] Unable to initialize EyeLink.\n"; + exit( EXIT_FAILURE ); } + run_trials(); } int main() diff --git a/src/frontend/gazedraw.cc b/src/frontend/gazedraw.cc new file mode 100644 index 0000000..966dca6 --- /dev/null +++ b/src/frontend/gazedraw.cc @@ -0,0 +1,171 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "cairo_objects.hh" +#include "display.hh" + +using namespace std; +using namespace std::chrono; + +void program_body() +{ + VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution + + Cairo cairo { 1920, 1080 }; + Pango pango { cairo }; + + /* open the PNG */ + PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/testim.png" }; + + /* draw gray over everything */ + cairo_new_path( cairo ); + cairo_identity_matrix( cairo ); + cairo_rectangle( cairo, 500, 500, 100, 100 ); + cairo_set_source_rgba( cairo, 0, 0.9, 0, 0.5 ); + cairo_fill( cairo ); + + /* draw the PNG */ + cairo_identity_matrix( cairo ); + cairo_scale( cairo, 0.8, 0.8 ); + double center_x = 960, center_y = 540; + cairo_device_to_user( cairo, ¢er_x, ¢er_y ); + cairo_translate( cairo, center_x, center_y ); + cairo_set_source_surface( cairo, png_image, 0, 0 ); + cairo_paint( cairo ); + + /* draw some text */ + Pango::Font myfont { "Times New Roman, 80" }; + Pango::Text mystring { cairo, pango, myfont, "Hello, world, Brooke, and Luke." }; + mystring.draw_centered_at( cairo, 960, 540 ); + cairo_set_source_rgba( cairo, 1, 1, 1, 1 ); + cairo_fill( cairo ); + + /* finish and copy to YUV raster */ + cairo.flush(); + + unsigned int stride = cairo.stride(); + Raster420 yuv_raster { 1920, 1080 }; + for ( unsigned int y = 0; y < 1080; y++ ) { + for ( unsigned int x = 0; x < 1920; x++ ) { + yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )]; + } + } + + Texture420 texture { yuv_raster }; + display.draw( texture ); + pause(); +} + +void writeTextRaster(Raster420 & yuv_raster, float pos_x, float pos_y) { + Cairo cairo { 1920, 1080 }; + Pango pango { cairo }; + cairo_new_path( cairo ); + + Pango::Font myfont { "Times New Roman, 80" }; + Pango::Text mystring { cairo, pango, myfont, "Eye" }; + mystring.draw_centered_at( cairo, pos_x * 1920, pos_y * 1080); + cairo_set_source_rgba( cairo, 1, 1, 1, 1 ); + cairo_fill( cairo ); + cairo.flush(); + + unsigned int stride = cairo.stride(); + for ( unsigned int y = 0; y < 1080; y++ ) { + for ( unsigned int x = 0; x < 1920; x++ ) { + yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )]; + } + } + +} + + + +int gazeUpdate(float & pos_x) +{ + zmq::context_t context; + + zmq::socket_t sock(context, zmq::socket_type::req); + fprintf(stderr, "Connecting to socket.\n"); + sock.connect("tcp://127.0.0.1:4587"); + fprintf(stderr, "Connected.\n"); + //sock.send(zmq::str_buffer("R"), zmq::send_flags::dontwait); + // fprintf(stderr, "Sent.\n"); + + fprintf(stderr, "Send SUB_PORT.\n"); + sock.send(zmq::str_buffer("SUB_PORT"), zmq::send_flags::dontwait); + zmq::message_t sub_port; + fprintf(stderr, "Recv SUB_PORT.\n"); + auto ret = sock.recv(sub_port, zmq::recv_flags::none); + if (!ret) + return 1; + cout << "SUB_PORT: " << sub_port.to_string() << "\n"; + + sock.send(zmq::str_buffer("PUB_PORT"), zmq::send_flags::dontwait); + zmq::message_t pub_port; + ret = sock.recv(pub_port, zmq::recv_flags::none); + if (!ret) + return 1; + cout << "PUB_PORT: " << pub_port.to_string() << "\n"; + + zmq::socket_t subscriber(context, zmq::socket_type::sub); + subscriber.connect("tcp://127.0.0.1:" + sub_port.to_string()); + + subscriber.setsockopt(ZMQ_SUBSCRIBE, "gaze", strlen("gaze")); + + while (true) { + vector recv_msgs; + ret = zmq::recv_multipart(subscriber, std::back_inserter(recv_msgs)); + if (!ret) + return 1; + + msgpack::object_handle oh = msgpack::unpack((const char*) recv_msgs[1].data(), recv_msgs[1].size()); + msgpack::object obj = oh.get(); + //cout << obj << endl; + //string eye = obj.via.array.ptr[1].as(); + //cout << "eye: " << eye[8] << endl; + //float timestamp = obj.via.array.ptr[7].as(); + //cout << "timestamp: " << timestamp << endl; + float pos1 = obj.via.array.ptr[3].via.array.ptr[0].as(); + //float pos2 = obj.via.array.ptr[3].via.array.ptr[1].as(); + //cout << "pos: " << "[" << pos1 << ", " << pos2 << "]" << endl; + if (!isnan(pos1)) { + pos_x = pos1; + } + } + return 0; +} + +void test() { + VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution + Raster420 yuv_raster { 1920, 1080 }; + float pos_x; + gazeUpdate(pos_x); + writeTextRaster(yuv_raster, pos_x, 0.5); + Texture420 texture { yuv_raster }; + display.draw( texture ); + pause(); +} + +int main() +{ + try { + //program_body(); + test(); + } catch ( const exception& e ) { + cerr << "Exception: " << e.what() << "\n"; + return EXIT_FAILURE; + } + + return EXIT_SUCCESS; +} diff --git a/src/frontend/pupil_labs.cc b/src/frontend/pupil_labs.cc new file mode 100644 index 0000000..fd7a336 --- /dev/null +++ b/src/frontend/pupil_labs.cc @@ -0,0 +1,64 @@ +#include +#include +#include +#include +#include + +#include +#include +#include + +using namespace std; + +int main() +{ + zmq::context_t context; + + zmq::socket_t sock(context, zmq::socket_type::req); + fprintf(stderr, "Connecting to socket.\n"); + sock.connect("tcp://127.0.0.1:4587"); + fprintf(stderr, "Connected.\n"); + //sock.send(zmq::str_buffer("R"), zmq::send_flags::dontwait); + // fprintf(stderr, "Sent.\n"); + + fprintf(stderr, "Send SUB_PORT.\n"); + sock.send(zmq::str_buffer("SUB_PORT"), zmq::send_flags::dontwait); + zmq::message_t sub_port; + fprintf(stderr, "Recv SUB_PORT.\n"); + auto ret = sock.recv(sub_port, zmq::recv_flags::none); + if (!ret) + return 1; + cout << "SUB_PORT: " << sub_port.to_string() << "\n"; + + sock.send(zmq::str_buffer("PUB_PORT"), zmq::send_flags::dontwait); + zmq::message_t pub_port; + ret = sock.recv(pub_port, zmq::recv_flags::none); + if (!ret) + return 1; + cout << "PUB_PORT: " << pub_port.to_string() << "\n"; + + zmq::socket_t subscriber(context, zmq::socket_type::sub); + subscriber.connect("tcp://127.0.0.1:" + sub_port.to_string()); + + subscriber.setsockopt(ZMQ_SUBSCRIBE, "gaze", strlen("gaze")); + + while (true) { + vector recv_msgs; + ret = zmq::recv_multipart(subscriber, std::back_inserter(recv_msgs)); + if (!ret) + return 1; + + msgpack::object_handle oh = msgpack::unpack((const char*) recv_msgs[1].data(), recv_msgs[1].size()); + msgpack::object obj = oh.get(); + //cout << obj << endl; + string eye = obj.via.array.ptr[1].as(); + cout << "eye: " << eye[8] << endl; + float timestamp = obj.via.array.ptr[7].as(); + cout << "timestamp: " << timestamp << endl; + float pos1 = obj.via.array.ptr[3].via.array.ptr[0].as(); + float pos2 = obj.via.array.ptr[3].via.array.ptr[1].as(); + cout << "pos: " << "[" << pos1 << ", " << pos2 << "]" << endl; + } + + return 0; +} diff --git a/src/frontend/user_test.cc b/src/frontend/user_test.cc new file mode 100644 index 0000000..3956da1 --- /dev/null +++ b/src/frontend/user_test.cc @@ -0,0 +1,346 @@ +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include + +#include "display.hh" + +#define CURSOR_SIZE 5 /* radius of the white dot in px */ +#define NUM_TRIALS 1 + +using namespace std; +using namespace std::chrono; + +/** + * End recording: adds 100 msec of data to catch final events + */ +static void end_trial( void ) +{ + pump_delay( 100 ); // provide a small amount of delay for last data + stop_recording(); + while ( getkey() ) { + }; +} + +int get_tracker_sw_version( char* verstr ) +{ + int ln = 0; + int st = 0; + ln = strlen( verstr ); + while ( ln > 0 && verstr[ln - 1] == ' ' ) + verstr[--ln] = 0; // trim + + // find the start of the version number + st = ln; + while ( st > 0 && verstr[st - 1] != ' ' ) + st--; + return atoi( &verstr[st] ); +} + +int initialize_eyelink() +{ + char verstr[50]; + int eyelink_ver = 0; + int tracker_software_ver = 0; + + // Set the address of the tracker, this is hard-coded by the Eyelink + set_eyelink_address( (char*)"100.1.1.1" ); + + // Initialize the EyeLink DLL and connect to the tracker + // * 0 opens a connection with the eye tracker + // * 1 will create a dummy connection for simulation + // * -1 initializes the DLL but does not open a connection + if ( open_eyelink_connection( 0 ) ) + return -1; + + set_offline_mode(); + flush_getkey_queue(); + + // Now configure tracker for display resolution + eyecmd_printf( "screen_pixel_coords = %ld %ld %ld %ld", 0, 0, 1920, 1080 ); + + eyelink_ver = eyelink_get_tracker_version( verstr ); + if ( eyelink_ver == 3 ) + tracker_software_ver = get_tracker_sw_version( verstr ); + + // SET UP TRACKER CONFIGURATION + // set parser saccade thresholds (conservative settings) + if ( eyelink_ver >= 2 ) { + eyecmd_printf( "select_parser_configuration 0" ); // 0 = standard sensitivity + // turn off scenelink camera stuff + if ( eyelink_ver == 2 ) { + eyecmd_printf( "scene_camera_gazemap = NO" ); + } + } else { + eyecmd_printf( "saccade_velocity_threshold = 35" ); + eyecmd_printf( "saccade_acceleration_threshold = 9500" ); + } + + // set link data (used for gaze cursor) + eyecmd_printf( "link_event_filter = LEFT,RIGHT,FIXATION,SACCADE,BLINK,BUTTON,INPUT" ); + eyecmd_printf( "link_sample_data = LEFT,RIGHT,GAZE,GAZERES,AREA,STATUS%s,INPUT", + ( tracker_software_ver >= 4 ) ? ",HTARGET" : "" ); + + // Make sure we're still alive + if ( !eyelink_is_connected() || break_pressed() ) { + return -1; + } + return 0; +} + +int exit_eyelink() +{ + close_expt_graphics(); // tell EXPTSPPT to release window + close_eyelink_connection(); // disconnect from tracker + return 0; +} + +void clear_full_screen_window( SDL_Surface* window, SDL_Color c ) +{ + SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) ); + SDL_Flip( window ); + SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) ); +} + +void do_calibration() +{ + // The colors of the target and background for calibration and drift correct + SDL_Color target_background_color = { 192, 192, 192, 0 }; + SDL_Color target_foreground_color = { 0, 0, 0, 0 }; + SDL_Surface* window = NULL; + + // register window with EXPTSPPT + if ( init_expt_graphics( NULL, NULL ) ) { + exit_eyelink(); + return; + } + + // Setup calibration type + eyecmd_printf( "calibration_type = HV9" ); + + window = SDL_GetVideoSurface(); + + // Size for calibration target and focal spot + unsigned int i = 1920 / 60; + unsigned int j = 1920 / 300; + if ( j < 2 ) + j = 2; + set_target_size( i, j ); // tell DLL the size of target features + + // tell EXPTSPPT the colors + set_calibration_colors( &target_foreground_color, &target_background_color ); + + clear_full_screen_window( window, target_background_color ); + + SDL_Flip( window ); + + do_tracker_setup(); + + // Wait for user to press escape before continuing + while ( !escape_pressed() ) { + }; + + close_expt_graphics(); // tell EXPTSPPT to release window +} + +int gc_window_trial() +{ + ofstream log; + log.open( "results.csv" ); + unsigned int frame_count = 0; + int button; /* the button pressed (0 if timeout) */ + + // First, set up all the textures + VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution + display.window().hide_cursor( true ); + + // whether to wait for vertical retrace before swapping buffer + // * 0 for immediate updates + // * 1 for updates synchronized with the vertical retrace + // * -1 for adaptive vsync + display.window().set_swap_interval( 0 ); + + // Used to track gaze samples + ALLF_DATA evt; + float x_sample, y_sample; + + // Ensure Eyelink has enough time to switch modes + set_offline_mode(); + pump_delay( 50 ); + + // Start data streaming + // Note that we are ignoring the EDF file. + int error = start_recording( 0, 0, 1, 1 ); + if ( error != 0 ) { + log.close(); + return error; + } + + // wait for link sample data + if ( !eyelink_wait_for_block_start( 100, 1, 0 ) ) { + end_trial(); + cerr << "ERROR: No link samples received!\n"; + log.close(); + return TRIAL_ERROR; + } + + // determine which eye(s) are available + int eye_used = eyelink_eye_available(); + + // reset keys and buttons from tracker + eyelink_flush_keybuttons( 0 ); + + const auto start_time = steady_clock::now(); + + // Poll for new samples until the diff between samples is large enough to signify LEDs switched + while ( true ) { + // Termination conditions + if ( ( error = check_recording() ) != 0 ) { + log.close(); + return error; + } + + if ( break_pressed() ) /* check for program termination or ALT-F4 or CTRL-C keys */ + { + end_trial(); /* local function to stop recording */ + log.close(); + return ABORT_EXPT; /* return this code to terminate experiment */ + } + + if ( escape_pressed() ) /* check for local ESC key to abort trial (useful in debugging) */ + { + end_trial(); /* local function to stop recording */ + log.close(); + return SKIP_TRIAL; /* return this code if trial terminated */ + } + + /* BUTTON RESPONSE TEST */ + /* Check for eye-tracker buttons pressed */ + /* This is the preferred way to get response data or end trials */ + button = eyelink_last_button_press( NULL ); + if ( button != 0 ) /* button number, or 0 if none pressed */ + { + end_trial(); /* local function to stop recording */ + break; /* exit trial loop */ + } + + // check for new sample update + if ( eyelink_newest_float_sample( NULL ) > 0 ) { + eyelink_newest_float_sample( &evt ); + + x_sample = evt.fs.gx[eye_used]; + y_sample = evt.fs.gy[eye_used]; + + const auto t_sample = duration_cast( steady_clock::now() - start_time ).count(); + log << t_sample << ", " << x_sample << ", " << y_sample << endl; + + // make sure pupil is present + if ( x_sample != MISSING_DATA && y_sample != MISSING_DATA && evt.fs.pa[eye_used] > 0 ) { + // TODO: Draw a dot where we are looking. This is currently very naive, + // we should be able to draw a dot faster. Right now, this is limiting + // our FPS to about 77fps. + + /* draw cursor location (235 = max luma in typical Y'CbCr colorspace) */ + Raster420 cursor { 1920, 1080 }; + + for ( unsigned int y = 0; y < cursor.Y.height(); y++ ) { + for ( unsigned int x = 0; x < cursor.Y.width(); x++ ) { + // Check if point is within radius + bool within_circle = ( ( ( x_sample - x ) * ( x_sample - x ) ) + + ( ( y_sample - y ) * ( y_sample - y ) ) ) <= ( CURSOR_SIZE * CURSOR_SIZE ); + const uint8_t color = within_circle ? 235 : 16; + cursor.Y.at( x, y ) = color; + } + } + + Texture420 cursor_texture { cursor }; + + // Draw texture. Note this may introduce extra delay since this is + // computing a texture each update. + display.draw( cursor_texture ); + + frame_count++; + + if ( frame_count % 480 == 0 ) { + const auto now = steady_clock::now(); + const auto ms_elapsed = duration_cast( now - start_time ).count(); + cout << "Drew " << frame_count << " frames in " << ms_elapsed + << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n"; + } + } + } + } + end_trial(); + log.close(); + return check_record_exit(); +} + +int run_trials() +{ + + do_calibration(); + + for ( unsigned int trial = 0; trial < NUM_TRIALS; trial++ ) { + // abort if link is closed + if ( eyelink_is_connected() == 0 || break_pressed() ) { + return ABORT_EXPT; + } + + int i = gc_window_trial(); + + // Report errors + switch ( i ) { + case ABORT_EXPT: // handle experiment abort or disconnect + cout << "EXPERIMENT ABORTED\n"; + return ABORT_EXPT; + case REPEAT_TRIAL: // trial restart requested + cout << "TRIAL REPEATED\n"; + trial--; + break; + case SKIP_TRIAL: // skip trial + cout << "TRIAL ABORTED\n"; + break; + case TRIAL_OK: // successful trial + cout << "TRIAL OK\n"; + break; + default: // other error code + cout << "TRIAL ERROR\n"; + break; + } + } + + return 0; +} + +void program_body() +{ + if ( initialize_eyelink() < 0 ) { + cerr << "[Error] Unable to initialize EyeLink.\n"; + exit( EXIT_FAILURE ); + } + run_trials(); +} + +int main() +{ + try { + program_body(); + } catch ( const exception& e ) { + cerr << "Exception: " << e.what() << "\n"; + return EXIT_FAILURE; + } + + return EXIT_SUCCESS; +} diff --git a/src/util/Makefile.am b/src/util/Makefile.am index c010a78..977d008 100644 --- a/src/util/Makefile.am +++ b/src/util/Makefile.am @@ -1,6 +1,7 @@ -AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLFW3_CFLAGS) $(GLEW_CFLAGS) +AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLFW3_CFLAGS) $(GLEW_CFLAGS) $(PANGOCAIRO_CFLAGS) AM_CXXFLAGS = $(PICKY_CXXFLAGS) noinst_LIBRARIES = libgldemoutil.a -libgldemoutil_a_SOURCES = gl_objects.hh gl_objects.cc display.hh display.cc +libgldemoutil_a_SOURCES = gl_objects.hh gl_objects.cc display.hh display.cc \ + cairo_objects.hh cairo_objects.cc diff --git a/src/util/cairo_objects.cc b/src/util/cairo_objects.cc new file mode 100644 index 0000000..1288ea8 --- /dev/null +++ b/src/util/cairo_objects.cc @@ -0,0 +1,181 @@ +#include "cairo_objects.hh" + +#include +#include + +using namespace std; + +Surface::Surface( cairo_surface_t* surface_ptr ) + : surface_( surface_ptr ) +{ + if ( not surface_ptr ) { + throw runtime_error( "surface_ptr is null" ); + } + + check_error(); +} + +void Surface::Deleter::operator()( cairo_surface_t* x ) const +{ + cairo_surface_destroy( x ); +} + +void Cairo::Context::Deleter::operator()( cairo_t* x ) const +{ + cairo_destroy( x ); +} + +void Cairo::Pattern::Deleter::operator()( cairo_pattern_t* x ) const +{ + cairo_pattern_destroy( x ); +} + +void Pango::Font::Deleter::operator()( PangoFontDescription* x ) const +{ + pango_font_description_free( x ); +} + +void Pango::Text::Deleter::operator()( cairo_path_t* x ) const +{ + cairo_path_destroy( x ); +} + +Cairo::Cairo( const unsigned int width, const unsigned int height ) + : surface_( width, height ) + , context_( surface_ ) +{ + check_error(); +} + +ImageSurface::ImageSurface( cairo_surface_t* surface_ptr ) + : Surface( surface_ptr ) + , width_( cairo_image_surface_get_width( *this ) ) + , height_( cairo_image_surface_get_height( *this ) ) + , stride_( cairo_image_surface_get_stride( *this ) ) +{ + check_error(); +} + +Cairo::Context::Context( ImageSurface& surface ) + : context( cairo_create( surface ) ) +{ + check_error(); +} + +void Surface::check_error() +{ + const cairo_status_t surface_result = cairo_surface_status( surface_.get() ); + if ( surface_result ) { + throw runtime_error( string( "cairo surface error: " ) + cairo_status_to_string( surface_result ) ); + } +} + +void Cairo::Context::check_error() +{ + const cairo_status_t context_result = cairo_status( context.get() ); + if ( context_result ) { + throw runtime_error( string( "cairo context error: " ) + cairo_status_to_string( context_result ) ); + } +} + +void Cairo::check_error() +{ + context_.check_error(); + surface_.check_error(); +} + +Pango::Pango( Cairo& cairo ) + : context_( pango_cairo_create_context( cairo ) ) + , layout_( pango_layout_new( *this ) ) +{} + +Pango::Font::Font( const string& description ) + : font( pango_font_description_from_string( description.c_str() ) ) +{} + +void Pango::set_font( const Pango::Font& font ) +{ + pango_layout_set_font_description( *this, font ); +} + +mutex& global_pango_mutex() +{ + static mutex global_pango_mutex_; + + return global_pango_mutex_; +} + +Pango::Text::Text( Cairo& cairo, Pango& pango, const Font& font, const string& text ) + : path_() + , extent_( { 0, 0, 0, 0 } ) +{ + unique_lock ul { global_pango_mutex() }; + + cairo_identity_matrix( cairo ); + cairo_new_path( cairo ); + + pango.set_font( font ); + + pango_layout_set_markup( pango, text.data(), text.size() ); + + pango_cairo_layout_path( cairo, pango ); + + path_.reset( cairo_copy_path( cairo ) ); + + /* get logical extents */ + PangoRectangle logical; + pango_layout_get_extents( pango, nullptr, &logical ); + extent_ = { logical.x / double( PANGO_SCALE ), + logical.y / double( PANGO_SCALE ), + logical.width / double( PANGO_SCALE ), + logical.height / double( PANGO_SCALE ) }; +} + +void Pango::Text::draw_centered_at( Cairo& cairo, const double x, const double y, const double max_width ) const +{ + cairo_identity_matrix( cairo ); + cairo_new_path( cairo ); + Cairo::Extent my_extent = extent().to_device( cairo ); + + double center_x = x - my_extent.x - my_extent.width / 2; + double center_y = y - my_extent.y - my_extent.height / 2; + + if ( my_extent.width > max_width ) { + const double scale_factor = max_width / my_extent.width; + cairo_scale( cairo, scale_factor, scale_factor ); + center_x = x - my_extent.x - my_extent.width * scale_factor / 2; + center_y = y - my_extent.y - my_extent.height * scale_factor / 2; + } + + cairo_device_to_user( cairo, ¢er_x, ¢er_y ); + + cairo_translate( cairo, center_x, center_y ); + + cairo_append_path( cairo, path_.get() ); +} + +void Pango::Text::draw_centered_rotated_at( Cairo& cairo, const double x, const double y ) const +{ + cairo_identity_matrix( cairo ); + cairo_new_path( cairo ); + + cairo_rotate( cairo, -3.1415926 / 2.0 ); + + Cairo::Extent my_extent = extent().to_device( cairo ); + + double center_x = x - my_extent.x - my_extent.width / 2; + double center_y = y - my_extent.y - my_extent.height / 2; + + cairo_device_to_user( cairo, ¢er_x, ¢er_y ); + cairo_translate( cairo, center_x, center_y ); + cairo_append_path( cairo, path_.get() ); +} + +Cairo::Pattern::Pattern( cairo_pattern_t* pattern ) + : pattern_( pattern ) +{ + const cairo_status_t pattern_result = cairo_pattern_status( pattern_.get() ); + if ( pattern_result ) { + throw runtime_error( string( "cairo pattern error: " ) + cairo_status_to_string( pattern_result ) ); + } +} diff --git a/src/util/cairo_objects.hh b/src/util/cairo_objects.hh new file mode 100644 index 0000000..68c94a7 --- /dev/null +++ b/src/util/cairo_objects.hh @@ -0,0 +1,198 @@ +#ifndef CAIRO_OBJECTS_HH +#define CAIRO_OBJECTS_HH + +#include +#include +#include +#include + +#include "gl_objects.hh" + +class Surface +{ + struct Deleter + { + void operator()( cairo_surface_t* x ) const; + }; + + std::unique_ptr surface_; + +protected: + Surface( cairo_surface_t* surface_ptr ); + +public: + void check_error(); + + operator cairo_surface_t*() { return surface_.get(); } + operator const cairo_surface_t*() const { return surface_.get(); } +}; + +class ImageSurface : public Surface +{ + unsigned int width_, height_, stride_; + +protected: + ImageSurface( cairo_surface_t* surface_ptr ); + +public: + uint8_t* pixels() { return cairo_image_surface_get_data( *this ); } + + unsigned int width() const { return width_; } + unsigned int height() const { return height_; } + unsigned int stride() const { return stride_; } +}; + +class FreshImageSurface : public ImageSurface +{ +public: + FreshImageSurface( const unsigned int width, const unsigned int height ) + : ImageSurface( cairo_image_surface_create( CAIRO_FORMAT_RGB24, width, height ) ) + {} +}; + +class PNGSurface : public ImageSurface +{ +public: + PNGSurface( const char* filename ) + : ImageSurface( cairo_image_surface_create_from_png( filename ) ) + {} +}; + +class Cairo +{ + FreshImageSurface surface_; + + struct Context + { + struct Deleter + { + void operator()( cairo_t* x ) const; + }; + + std::unique_ptr context; + + Context( ImageSurface& surface ); + + void check_error(); + } context_; + + void check_error(); + +public: + Cairo( const unsigned int width, const unsigned int height ); + + unsigned int width() { return surface_.width(); } + unsigned int height() { return surface_.height(); } + unsigned int stride() { return surface_.stride(); } + + operator cairo_t*() { return context_.context.get(); } + + uint8_t* pixels() { return surface_.pixels(); } + void flush() { cairo_surface_flush( surface_ ); } + + template + struct Extent + { + double x, y, width, height; + + Extent to_user( Cairo& cairo ) const + { + static_assert( device_coordinates == true, "Extent::to_user() called but coordinates already in user-space" ); + + double x1 = x, x2 = x + width, y1 = y, y2 = y + height; + + cairo_device_to_user( cairo, &x1, &y1 ); + cairo_device_to_user( cairo, &x2, &y2 ); + + return Extent( { x1, y1, x2 - x1, y2 - y1 } ); + } + + Extent to_device( Cairo& cairo ) const + { + static_assert( device_coordinates == false, + "Extent::to_device() called but coordinates already in device-space" ); + + double x1 = x, x2 = x + width, y1 = y, y2 = y + height; + + cairo_user_to_device( cairo, &x1, &y1 ); + cairo_user_to_device( cairo, &x2, &y2 ); + + return Extent( { x1, y1, x2 - x1, y2 - y1 } ); + } + }; + + class Pattern + { + struct Deleter + { + void operator()( cairo_pattern_t* x ) const; + }; + + std::unique_ptr pattern_; + + public: + Pattern( cairo_pattern_t* pattern ); + + operator cairo_pattern_t*() { return pattern_.get(); } + }; +}; + +template +struct PangoDelete +{ + void operator()( T* x ) const { g_object_unref( x ); } +}; + +class Pango +{ + std::unique_ptr> context_; + std::unique_ptr> layout_; + +public: + Pango( Cairo& cairo ); + + operator PangoContext*() { return context_.get(); } + operator PangoLayout*() { return layout_.get(); } + + struct Font + { + struct Deleter + { + void operator()( PangoFontDescription* x ) const; + }; + + std::unique_ptr font; + + Font( const std::string& description ); + + operator const PangoFontDescription*() const { return font.get(); } + }; + + class Text + { + struct Deleter + { + void operator()( cairo_path_t* x ) const; + }; + + std::unique_ptr path_; + Cairo::Extent extent_; + + public: + Text( Cairo& cairo, Pango& pango, const Font& font, const std::string& text ); + + const Cairo::Extent& extent() const { return extent_; } + + void draw_centered_at( Cairo& cairo, + const double x, + const double y, + const double max_width = std::numeric_limits::max() ) const; + void draw_centered_rotated_at( Cairo& cairo, const double x, const double y ) const; + + operator const cairo_path_t*() const { return path_.get(); } + }; + + void set_font( const Font& font ); +}; + +#endif /* CAIRO_OBJECTS_HH */ diff --git a/src/util/display.cc b/src/util/display.cc index b3dd897..8da6976 100644 --- a/src/util/display.cc +++ b/src/util/display.cc @@ -36,14 +36,14 @@ const string VideoDisplay::shader_source_scale_from_pixel_coordinates = R"( #ver in vec2 position; in vec2 chroma_texcoord; - out vec2 raw_position; + out vec2 Y_texcoord; out vec2 uv_texcoord; void main() { gl_Position = vec4( 2 * position.x / window_size.x - 1.0, 1.0 - 2 * position.y / window_size.y, 0.0, 1.0 ); - raw_position = vec2( position.x, position.y ); + Y_texcoord = vec2( position.x, position.y ); uv_texcoord = vec2( chroma_texcoord.x, chroma_texcoord.y ); } )"; @@ -70,15 +70,59 @@ const string VideoDisplay::shader_source_ycbcr = R"( #version 130 uniform sampler2DRect uTex; uniform sampler2DRect vTex; + uniform vec3 head_orientation; + + in vec2 Y_texcoord; in vec2 uv_texcoord; - in vec2 raw_position; - out vec4 outColor; + out vec4 outColor; + + mat3 eul2rotm( float rotX, float rotY, float rotZ ) { + + mat3 R_x = mat3( 1.0f, 0f, 0f, + 0f, cos(rotX), sin(rotX), + 0f, -sin(rotX), cos(rotX)); + mat3 R_y = mat3( cos(rotY), 0f, -sin(rotY), + 0f, 1.0f, 0f, + sin(rotY), 0f, cos(rotY)); + mat3 R_z = mat3( cos(rotZ), sin(rotZ), 0f, + -sin(rotZ), cos(rotZ), 0f, + 0f, 0f, 1.0f); + mat3 invCamMat = mat3(0.0015f, 0f, 0f, + 0f, 0.0015f, 0f, + -1.4286f, -0.8036f, 1f); + + return R_z * R_y * R_x * invCamMat; + } + + vec2 get_latlong( vec2 Y_texcoord ) { + vec3 xyz = vec3( Y_texcoord.x, Y_texcoord.y, 1.0 ); + vec3 xyz_norm = xyz / length(xyz); + + vec3 ray3d = eul2rotm(head_orientation.x, head_orientation.y, head_orientation.z) * xyz_norm; + float theta = atan( ray3d.y, length(ray3d.xz) ); + float phi = atan( ray3d.x, ray3d.z ); + return vec2( phi, theta); + } + + vec2 reproject_Y (vec2 Y_texcoord ) { + vec2 phi_theta = get_latlong(Y_texcoord); + vec2 xy_sphere = vec2( ((phi_theta.x / 3.14f) * 1920f + 1920.0)/2.0, (phi_theta.y + 3.14/2.0) * 1080f /3.14 ); + return xy_sphere; + } + + vec2 reproject_uv( vec2 Y_texcoord ) { + vec2 phi_theta = get_latlong(Y_texcoord); + vec2 xy_sphere = vec2( ((phi_theta.x / 3.14f) * 960f + 960.0)/2.0, (phi_theta.y + 3.14/2.0) * 540f /3.14 ); + return xy_sphere; + } void main() { - float fY = texture(yTex, raw_position).x; - float fCb = texture(uTex, uv_texcoord).x; - float fCr = texture(vTex, uv_texcoord).x; + vec2 Y_reprojected = reproject_Y(Y_texcoord); + vec2 uv_reprojected = reproject_uv(Y_texcoord) + max( 0.0, min( 0.0, texture(uTex, uv_texcoord).x ) ); // to get rid of inefficiency bug + float fY = texture(yTex, Y_reprojected).x; + float fCb = texture(uTex, uv_reprojected).x; + float fCr = texture(vTex, uv_reprojected).x; outColor = vec4( max(0, min(1.0, 1.16438356164384 * (fY - 0.06274509803921568627) + 1.59567019581339 * (fCr - 0.50196078431372549019))), @@ -128,6 +172,12 @@ VideoDisplay::VideoDisplay( const unsigned int width, const unsigned int height, glCheck( "VideoDisplay constructor" ); } +void VideoDisplay::update_head_orientation( const float x, const float y, const float z ) +{ + texture_shader_program_.use(); + glUniform3f( texture_shader_program_.uniform_location( "head_orientation" ), x, y, z ); +} + void VideoDisplay::resize( const unsigned int width, const unsigned int height ) { glViewport( 0, 0, width, height ); @@ -167,6 +217,12 @@ void VideoDisplay::resize( const unsigned int width, const unsigned int height ) glCheck( "after installing shaders" ); } +// void VideoDisplay::updateHeadOrientation( const float pitch, const float roll, const float yaw) { +// texture_shader_program_.use(); +// glUniform3f( texture_shader_program_.uniform_location( "head_orientation" ), pitch, roll, yaw ); +// } + + void VideoDisplay::draw( Texture420& image ) { image.bind(); diff --git a/src/util/display.hh b/src/util/display.hh index e3aab0a..561ea8f 100644 --- a/src/util/display.hh +++ b/src/util/display.hh @@ -68,9 +68,13 @@ public: void repaint(); void resize( const unsigned int width, const unsigned int height ); + //void updateHeadOrientation( const float pitch, const float roll, const float yaw); + Window& window() { return current_context_window_.window_; } const Window& window() const { return current_context_window_.window_; } + void update_head_orientation( const float x, const float y, const float z ); + /* forbid copying */ VideoDisplay( const VideoDisplay& other ) = delete; VideoDisplay& operator=( const VideoDisplay& other ) = delete;