diff --git a/README.md b/README.md
index 108d9dd..75ec82c 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,53 @@
-OpenGL demo
-===========
+# EyeLink Latency Characterization
-To install compiler and library dependencies on Debian/Ubuntu:
+A set of simple hardware and software used to characterize end-to-end latency of
+an EyeLink 1000 eye tracker and the Xubuntu 18.04 graphics stack with
+compositing disabled.
-```
-$ sudo apt install build-essential autotools-dev autoconf automake libgl-dev libglfw3-dev libglew-dev libglu-dev
-```
+## Getting Started
+
+### Prerequisites
+
+* **EyeLink Developers Kit for Linux**
+ Steps to install:
+
+ 1. Add signing key
+ ```
+ $ wget -O - "http://download.sr-support.com/software/dists/SRResearch/SRResearch_key" | sudo apt-key add -
+ ```
+ 2. Add apt repository
+ ```
+ $ sudo add-apt-repository "deb http://download.sr-support.com/software SRResearch main"
+ $ sudo apt-get update
+ ```
+ 3. Install latest release of EyeLink Developers Kit for Linux
+ ```
+ $ sudo apt-get install eyelink-display-software
+ ```
+ Alternatively, a tar of DEBs is available [at this link][debs].
+
+* **Network Configuration on Host PC**
+ The EyeLink PC and the Host PC communicate via a direct Ethernet connection
+ between the two computers. In order for third party programming
+ packages/languages to use this connection, you must use a static IP address for
+ the Ethernet port that is used to connect to the Host PC. This static IP should
+ be:
+
+ ```
+ IP Address: 100.1.1.2
+ Subnet Mask: 255.255.255.0
+ ```
+* **Host Software Dependencies**
+ To install compiler and library dependencies on Debian/Ubuntu:
+
+ ```
+ $ sudo apt install build-essential autotools-dev autoconf automake libgl-dev libglfw3-dev libglew-dev libglu-dev
+ ```
+* **Arduino IDE**
+ To install and set up the Arduino IDE, follow [this guide][arduino-guide], and
+ pay special attention to setting serial port permissions.
+
+### Host Software
To compile:
@@ -23,4 +65,130 @@ To use (from inside `build` directory):
$ ./src/frontend/example
```
-Main source code to read: [src/frontend/example.cc](https://github.com/keithw/gldemo/blob/master/src/frontend/example.cc)
+This will log the timing results to a file (results.csv) that can be analysed
+afterwords to visualize the latency distributions. The CSV file format is
+
+```
+e2e (us),eyelink (us),drawing (us)
+6852,1820,635
+9528,1616,595
+7180,1656,618
+5504,1194,632
+...
+```
+
+Main source code to read: [src/frontend/example.cc](src/frontend/example.cc).
+
+### Artificial Saccade Generator Software
+
+To setup the Arduino for use as the artificial saccade generator, use Arduino
+IDE on the Host computer to program the Arduino with the script found in
+[scripts/arduino.ino](scripts/arduino.ino).
+
+## Reference
+
+The following sections are information-oriented. They detail the specifics of
+the setup.
+
+### End-to-end Latency Measurement
+
+**Bill of Materials**
+
+* [1x] Artificial Saccade Generator (ASG), described below.
+* [1x] Computer running Xubuntu 18.04 with compositing DISABLED.
+* [1x] Dell P2815Q Display running 1920x1080 @ 240Hz using Zisworks x28 R2 kit.
+
+After the artificial saccade, we use a host computer to continuously monitor for
+gaze position changes. Once a change is detected, we use OpenGL to quickly
+switch a portion of the display white. A photodiode is mounted to the area of
+the display that will change. Once the photodiode triggers on the display
+change, we take the time difference between the display change event and
+toggling the IR LEDs as a measure of the end-to-end latency of the system. The
+sequence diagram below gives an overview of the process.
+
+```
+ ┌────┐ ┌───┐ ┌───────┐
+ │Host│ │ASG│ │Display│
+ └─┬──┘ └─┬─┘ └───┬───┘
+ │ Switch Eye Command │ │
+ │ ────────────────────────────────> │
+ │ │ │
+ │ ╔═══════════════╧════════════════╗ │
+ │ ║toggle IR LED, t_start = now() ░║ │
+ │ ╚═══════════════╤════════════════╝ │
+ │────┐ │
+ │ │ poll EyeLink for new samples │
+ │<───┘ │
+ │ │ │
+ │ │────┐
+ │ │ │ sample = photodiode level
+ │ │<───┘
+ │ │ │
+╔════════════╧═══════════════╗ │ │
+║new EyeLink Sample > DELTA ░║ │ │
+╚════════════╤═══════════════╝ │ │
+ │ draw white box │
+ │ ───────────────────────────────────────────────────────────────>
+ │ │ │
+ │ ╔══════════════════╧════════════════════╗ │
+ │ ║sample > trigger level, t_end = now() ░║ │
+ │ ╚══════════════════╤════════════════════╝ │
+ │ t_end - t_start │ │
+ │ <──────────────────────────────── │
+ ┌─┴──┐ ┌─┴─┐ ┌───┴───┐
+ │Host│ │ASG│ │Display│
+ └────┘ └───┘ └───────┘
+```
+
+A table clamp is used to secure the ASG to the table top so that it is easy to
+tune the resistance of the potentiometers, and pupil sizes (i.e., changing
+EyeLink settings, or printing different sized black dots on paper), such that
+the EyeLink detects the artificial pupil reliably. This is shown below. We also
+found that we needed to block some of the EyeLink's IR LED array so that the
+reflection off of the white paper did not hide the IR LEDs from the ASG.
+
+![setup][setup]
+
+### Artificial Saccade Generator
+
+To characterize the end-to-end latency, we use an artificial saccade generator,
+which works by tricking the eye tracker to detect an abrupt change in the gaze
+position of an artificial eye which does not move (a technique described in
+[[1]]). The key insight is that most video-based eye trackers compute gaze
+position by using the position of the pupil (emulated by a black dot laser
+printed on white copy paper) as well as the position of a corneal reflection
+(emulated by an infrared LED). By using two IR LEDs, we can programmatically
+trigger an artificial saccade by switching one IR LED off, while simultaneously
+switching the other IR LED on.
+
+We implement this using an Arduino Uno with a custom circuit soldered to a
+prototyping shield.
+
+**Bill of Materials**
+
+* [1x] F12N10L N-mosfet
+* [1x] 1RF9530 P-mosfet
+* [2x] Trim potentiometers
+* [1x] 2.2 M-ohm resistor
+* [1x] 22pF capacitor
+* [2x] Gikfun 5mm 940nm IR LEDs
+* [1x] FDS100 Photodiode
+* [1x] Mount + standoffs
+* [1x] Arduino Uno
+* [1x] Adafruit Proto Shield for Arduino
+* [1x] Black pupil laser printed on white paper
+
+The circuit that is soldered to the Arduino Proto Shield is shown below.
+
+![schematic][logo]
+
+The proto shield is then mounted to the Arduino, and the two are then mounted to
+an acrylic piece of plastic, with two holes cut to allow the IR LEDs to shine
+through from behind the artificial eye. The laser cutter SVG can be found in
+[docs/mount.svg](docs/mount.svg).
+
+[1]: https://www.ncbi.nlm.nih.gov/pubmed/24771998
+[arduino-guide]: https://www.arduino.cc/en/guide/linux
+[debs]: http://download.sr-support.com/linuxDisplaySoftwareRelease/eyelink-display-software_1.11_x64_debs.tar.gz
+[logo]: docs/circuit.png
+[setup]: docs/setup.jpg
diff --git a/configure.ac b/configure.ac
index 9f7c93a..14c4e65 100644
--- a/configure.ac
+++ b/configure.ac
@@ -28,6 +28,15 @@ PKG_CHECK_MODULES([GL], [gl])
PKG_CHECK_MODULES([GLU], [glu])
PKG_CHECK_MODULES([GLFW3], [glfw3])
PKG_CHECK_MODULES([GLEW], [glew])
+PKG_CHECK_MODULES([OPENCV], [opencv])
+PKG_CHECK_MODULES([PANGOCAIRO], [pangocairo])
+PKG_CHECK_MODULES([SDL], [sdl])
+PKG_CHECK_MODULES([SDL_GFX], [SDL_gfx])
+PKG_CHECK_MODULES([SDL_TTF], [SDL_ttf])
+PKG_CHECK_MODULES([SDL_IMAGE], [SDL_image])
+PKG_CHECK_MODULES([SDL_MIXER], [SDL_mixer])
+PKG_CHECK_MODULES([MSGPACK], [msgpack])
+PKG_CHECK_MODULES([LIBZMQ], [libzmq])
# Checks for header files.
AC_LANG_PUSH(C++)
diff --git a/docs/circuit.png b/docs/circuit.png
new file mode 100644
index 0000000..e8236fe
Binary files /dev/null and b/docs/circuit.png differ
diff --git a/docs/mount.svg b/docs/mount.svg
new file mode 100644
index 0000000..8bef1f1
--- /dev/null
+++ b/docs/mount.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/docs/setup.jpg b/docs/setup.jpg
new file mode 100644
index 0000000..1588b76
Binary files /dev/null and b/docs/setup.jpg differ
diff --git a/scripts/analysis/analysis.py b/scripts/analysis/analysis.py
new file mode 100755
index 0000000..86d1010
--- /dev/null
+++ b/scripts/analysis/analysis.py
@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+import argparse
+import logging
+import sys
+from subprocess import DEVNULL, run
+
+import matplotlib
+import matplotlib.pyplot as plt
+import pandas as pd
+import seaborn as sns
+import numpy as np
+from matplotlib.backends.backend_pdf import PdfPages
+
+matplotlib.rcParams["text.usetex"] = True
+
+sns.set(style="whitegrid")
+sns.set_context("paper", font_scale=1.5, rc={"lines.linewidth": 2.25})
+
+
+# Configure logging
+logging.basicConfig(
+ stream=sys.stdout,
+ format="[%(asctime)s][%(levelname)s] %(name)s:%(lineno)s - %(message)s",
+ level=logging.INFO,
+)
+logger = logging.getLogger(__name__)
+
+FIGSIZE = (7, 4)
+
+
+def _plot(infile):
+ """Plotting logic."""
+ fig, ax = plt.subplots(figsize=FIGSIZE)
+
+ # type,b,f1,precision,recall
+ data = pd.read_csv(infile, skipinitialspace=True)
+
+ # Plot PDF
+ plot = sns.distplot(
+ data["e2e (us)"] / 1000.0,
+ kde=False,
+ bins=25,
+ ax=ax
+ )
+
+ sns.despine(bottom=True, left=True)
+ plot.set(xlabel="End-to-end Latency (ms)")
+ plot.set(ylabel="Histogram")
+ outfile = "histogram.pdf"
+ pp = PdfPages(outfile)
+ pp.savefig(plot.get_figure().tight_layout())
+ pp.close()
+ # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True)
+ logger.info(f"Plot saved to {outfile}")
+
+ # Plot Boxplot
+ fig, ax = plt.subplots(figsize=FIGSIZE)
+ plot = sns.boxplot(
+ data = data,
+ orient = "h",
+ ax=ax
+ )
+
+ sns.despine(bottom=True, top=True)
+ plot.set(ylabel="Delay Type")
+ plot.set(xlabel="Time (us)")
+ outfile = "boxplot.pdf"
+ pp = PdfPages(outfile)
+ pp.savefig(plot.get_figure().tight_layout())
+ pp.close()
+ # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True)
+ logger.info(f"Plot saved to {outfile}")
+
+ # Plot CDF
+ fig, ax = plt.subplots(figsize=FIGSIZE)
+ plot = sns.distplot(
+ data["e2e (us)"] / 1000.0,
+ hist_kws={"cumulative": True, "rwidth": 0.85},
+ norm_hist=True,
+ # bins = 45,
+ kde=False
+ )
+
+ # handles, labels = ax.get_legend_handles_labels()
+ # ax.legend(handles=handles[1:], labels=labels[1:])
+ ax.set_ylim([0, 1])
+ # ax.set_xlim([0.5, 1])
+
+ sns.despine(bottom=True, left=True)
+ plot.set(xlabel="End-to-end Latency (ms)")
+ plot.set(ylabel="Cumulative Probability")
+ outfile = "cdf.pdf"
+ pp = PdfPages(outfile)
+ pp.savefig(plot.get_figure().tight_layout())
+ pp.close()
+ # run(["pdfcrop", outfile, outfile], stdout=DEVNULL, check=True)
+ logger.info(f"Plot saved to {outfile}")
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--data", type=str, default="../../results.csv", help="CSV file of latency data"
+ )
+ parser.add_argument(
+ "-v",
+ "--verbose",
+ dest="verbose",
+ action="store_true",
+ help="Output INFO level logging.",
+ )
+ args = parser.parse_args()
+
+ if args.verbose:
+ ch = logging.StreamHandler()
+ logger.setLevel(logging.INFO)
+ ch.setLevel(logging.INFO)
+ formatter = logging.Formatter("[%(levelname)s] %(name)s - %(message)s")
+ ch.setFormatter(formatter)
+ logger.addHandler(ch)
+
+ _plot(args.data)
diff --git a/scripts/analysis/requirements.txt b/scripts/analysis/requirements.txt
new file mode 100644
index 0000000..19bdfa0
--- /dev/null
+++ b/scripts/analysis/requirements.txt
@@ -0,0 +1,3 @@
+matplotlib
+pandas
+seaborn
diff --git a/scripts/arduino.ino b/scripts/arduino.ino
new file mode 100644
index 0000000..54fffc4
--- /dev/null
+++ b/scripts/arduino.ino
@@ -0,0 +1,56 @@
+// These pin numbers depend on which pins you use for your circuit
+int sensorPin = A0;
+int ledPin = 10;
+
+void setup()
+{
+ // Use a 115200 baud rate
+ Serial.begin(115200);
+ pinMode(ledPin, OUTPUT);
+ pinMode(sensorPin, INPUT);
+}
+
+void loop()
+{
+ static uint32_t ts1 = 0;
+ static uint32_t ts2 = 0;
+ static int state = 0;
+ static int sensorValue = 0;
+ int cmd = 0;
+
+ switch (state) {
+ case 0:
+ digitalWrite(ledPin, HIGH);
+
+ // poll for new command
+ if (Serial.available() > 0) {
+ cmd = Serial.read();
+
+ // command from the host PC is hard-coded as the character 'g'
+ if (cmd == 'g') {
+ // Start timer
+ ts1 = micros();
+
+ // Move to next state
+ state = 1;
+ }
+ }
+ break;
+
+ case 1:
+ digitalWrite(ledPin, LOW);
+ sensorValue = analogRead(sensorPin);
+
+ // Rising edge trigger condition
+ if (sensorValue > 512) {
+ // Log time difference and send back to host PC
+ ts2 = micros();
+ Serial.println(ts2 - ts1);
+
+ // Reset state
+ state = 0;
+ sensorValue = 0;
+ }
+ break;
+ }
+}
diff --git a/src/files/frame92.png b/src/files/frame92.png
new file mode 100644
index 0000000..e60a752
Binary files /dev/null and b/src/files/frame92.png differ
diff --git a/src/files/frame92_resized.png b/src/files/frame92_resized.png
new file mode 100644
index 0000000..64dace1
Binary files /dev/null and b/src/files/frame92_resized.png differ
diff --git a/src/files/testim.png b/src/files/testim.png
new file mode 100644
index 0000000..c16abb2
Binary files /dev/null and b/src/files/testim.png differ
diff --git a/src/frontend/Makefile.am b/src/frontend/Makefile.am
index 8ca7140..b2098b3 100644
--- a/src/frontend/Makefile.am
+++ b/src/frontend/Makefile.am
@@ -1,7 +1,22 @@
-AM_CPPFLAGS = $(CXX17_FLAGS) $(SSL_CFLAGS) -I$(srcdir)/../util
+AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLEW_CFLAGS) $(GLFW3_CFLAGS) $(PANGOCAIRO_CFLAGS) $(SDL_CFLAGS) $(SDL_GFX_CFLAGS) $(SDL_IMAGE_CFLAGS) $(SDL_TTF_CFLAGS) $(SDL_MIXER_CFLAGS) $(LIBZMQ_CFLAGS) $(MSGPACK_CFLAGS) $(OPENCV_CFLAGS) -I$(srcdir)/../util
AM_CXXFLAGS = $(PICKY_CXXFLAGS)
-bin_PROGRAMS = example
+bin_PROGRAMS = example drawtext user_test pupil_labs gazedraw equitest
example_SOURCES = example.cc
-example_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS)
+example_LDADD = -L/usr/lib -leyelink_core_graphics -leyelink_core -lsdl_util -lrt -leyelink_core_graphics -lpthread ../util/libgldemoutil.a $(SDL_LIBS) $(SDL_GFX_LIBS) $(SDL_IMAGE_LIBS) $(SDL_TTF_LIBS) $(SDL_MIXER_LIBS) $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS)
+
+drawtext_SOURCES = drawtext.cc
+drawtext_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS)
+
+user_test_SOURCES = user_test.cc
+user_test_LDADD = -L/usr/lib -leyelink_core_graphics -leyelink_core -lsdl_util -lrt -leyelink_core_graphics -lpthread ../util/libgldemoutil.a $(SDL_LIBS) $(SDL_GFX_LIBS) $(SDL_IMAGE_LIBS) $(SDL_TTF_LIBS) $(SDL_MIXER_LIBS) $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS)
+
+pupil_labs_SOURCES = pupil_labs.cc
+pupil_labs_LDADD = $(LIBZMQ_LIBS) $(MSGPACK_LIBS)
+
+gazedraw_SOURCES = gazedraw.cc
+gazedraw_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) $(LIBZMQ_LIBS) $(MSGPACK_LIBS)
+
+equitest_SOURCES = equitest.cc
+equitest_LDADD = ../util/libgldemoutil.a $(GLU_LIBS) $(GLEW_LIBS) $(GLFW3_LIBS) $(PANGOCAIRO_LIBS) $(OPENCV_LIBS)
\ No newline at end of file
diff --git a/src/frontend/drawtext.cc b/src/frontend/drawtext.cc
new file mode 100644
index 0000000..1bba2d8
--- /dev/null
+++ b/src/frontend/drawtext.cc
@@ -0,0 +1,74 @@
+#include
+#include
+#include
+#include
+#include
+
+#include
+
+#include "cairo_objects.hh"
+#include "display.hh"
+
+using namespace std;
+using namespace std::chrono;
+
+void program_body()
+{
+ VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution
+
+ Cairo cairo { 1920, 1080 };
+ Pango pango { cairo };
+
+ /* open the PNG */
+ PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/testim.png" };
+
+ /* draw gray over everything */
+ cairo_new_path( cairo );
+ cairo_identity_matrix( cairo );
+ cairo_rectangle( cairo, 500, 500, 100, 100 );
+ cairo_set_source_rgba( cairo, 0, 0.9, 0, 0.5 );
+ cairo_fill( cairo );
+
+ /* draw the PNG */
+ cairo_identity_matrix( cairo );
+ cairo_scale( cairo, 0.8, 0.8 );
+ double center_x = 960, center_y = 540;
+ cairo_device_to_user( cairo, ¢er_x, ¢er_y );
+ cairo_translate( cairo, center_x, center_y );
+ cairo_set_source_surface( cairo, png_image, 0, 0 );
+ cairo_paint( cairo );
+
+ /* draw some text */
+ Pango::Font myfont { "Times New Roman, 80" };
+ Pango::Text mystring { cairo, pango, myfont, "Hello, world, Brooke, and Luke." };
+ mystring.draw_centered_at( cairo, 960, 540 );
+ cairo_set_source_rgba( cairo, 1, 1, 1, 1 );
+ cairo_fill( cairo );
+
+ /* finish and copy to YUV raster */
+ cairo.flush();
+
+ unsigned int stride = cairo.stride();
+ Raster420 yuv_raster { 1920, 1080 };
+ for ( unsigned int y = 0; y < 1080; y++ ) {
+ for ( unsigned int x = 0; x < 1920; x++ ) {
+ yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )];
+ }
+ }
+
+ Texture420 texture { yuv_raster };
+ display.draw( texture );
+ pause();
+}
+
+int main()
+{
+ try {
+ program_body();
+ } catch ( const exception& e ) {
+ cerr << "Exception: " << e.what() << "\n";
+ return EXIT_FAILURE;
+ }
+
+ return EXIT_SUCCESS;
+}
diff --git a/src/frontend/equitest.cc b/src/frontend/equitest.cc
new file mode 100644
index 0000000..2b75317
--- /dev/null
+++ b/src/frontend/equitest.cc
@@ -0,0 +1,212 @@
+
+#include
+#include
+#include
+#include
+#include
+
+#include
+
+#include "cairo_objects.hh"
+#include "display.hh"
+
+#include "opencv2/core/core.hpp"
+#include "opencv2/highgui/highgui.hpp"
+#include "opencv2/imgproc/imgproc.hpp"
+
+using namespace std;
+using namespace std::chrono;
+
+#define SCREEN_RES_X 1920
+#define SCREEN_RES_Y 1080
+#define pi 3.14159265
+
+void writePNGRaster(Raster420 & yuv_raster) {
+ Cairo cairo { SCREEN_RES_X, SCREEN_RES_Y };
+
+ /* open the PNG */
+ PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/frame92_resized.png" };
+
+ /* draw the PNG */
+ cairo_identity_matrix( cairo );
+ //cairo_scale( cairo, 0.234375, 0.263671875 );
+ //cairo_scale( cairo, 0.5, 0.52734375 );
+ cairo_scale(cairo, 1.0, 1.0);
+ double center_x = 0, center_y = 0;
+ cairo_device_to_user( cairo, ¢er_x, ¢er_y );
+ cairo_translate( cairo, center_x, center_y );
+ cairo_set_source_surface( cairo, png_image, 0, 0 );
+ cairo_paint( cairo );
+
+ /* finish and copy to YUV raster */
+ cairo.flush();
+
+ unsigned int stride = cairo.stride();
+ for ( unsigned int y = 0; y < SCREEN_RES_Y; y++ ) {
+ for ( unsigned int x = 0; x < SCREEN_RES_X; x++ ) {
+ float red = cairo.pixels()[y * stride + 2 + ( x * 4 )] / 255.0;
+ float green = cairo.pixels()[y * stride + 1 + ( x * 4 )] / 255.0;
+ float blue = cairo.pixels()[y * stride + 0 + ( x * 4 )] / 255.0;
+
+ const float Ey = 0.7154 * green + 0.0721 * blue + 0.2125 * red;
+ const float Epb = -0.386 * green + 0.5000 * blue - 0.115 * red;
+ const float Epr = -0.454 * green - 0.046 * blue + 0.500 * red;
+
+ const uint8_t Y = (219 * Ey) + 16;
+ const uint8_t Cb = (224 * Epb) + 128;
+ const uint8_t Cr = (224 * Epr) + 128;
+
+ yuv_raster.Y.at( x, y ) = Y;
+ if ( (x%2) == 0 and (y%2) == 0 ) {
+ yuv_raster.Cb.at( x / 2, y / 2 ) = Cb;
+ yuv_raster.Cr.at( x / 2, y / 2 ) = Cr;
+ }
+ }
+ }
+}
+
+cv::Mat eul2rotm(double rotx, double roty, double rotz)
+{
+ cv::Mat R_x = (cv::Mat_(3, 3) << 1, 0, 0,
+ 0, cos(rotx), -sin(rotx),
+ 0, sin(rotx), cos(rotx));
+
+ cv::Mat R_y = (cv::Mat_(3, 3) << cos(roty), 0, sin(roty),
+ 0, 1, 0,
+ -sin(roty), 0, cos(roty));
+
+ cv::Mat R_z = (cv::Mat_(3, 3) << cos(rotz), -sin(rotz), 0,
+ sin(rotz), cos(rotz), 0,
+ 0, 0, 1);
+ cv::Mat K = (cv::Mat_(3, 3) << 672, 0, SCREEN_RES_X / 2,
+ 0, 672, SCREEN_RES_Y / 2,
+ 0, 0, 1);
+
+ cv::Mat R = R_z * R_y * R_x * K.inv();
+
+ return R;
+}
+
+cv::Mat xyz_norm_map_x;
+cv::Mat xyz_norm_map_y;
+
+std::pair reprojection(int x_img, int y_img, cv::Mat& img_src)
+{
+ cv::Mat xyz = (cv::Mat_(3, 1) << (double)x_img, (double)y_img, 1);
+ cv::Mat xyz_norm = xyz / norm(xyz);
+
+ //xyz_norm_map_x.at(x_img, y_img) = xyz_norm.at(0,0);
+
+ //std::cout << "[ " << xyz_norm.at(0, 0) << ", " <(0, 1) << ", " <(0, 2) << " ]" << std::endl;
+
+ cv::Mat Rot = eul2rotm(0.5, 0, 0);
+
+ //std::cout << Rot << std::endl;
+
+ cv::Mat ray3d = Rot * xyz_norm;
+
+ //get 3d spherical coordinates
+ double xp = ray3d.at(0, 0);
+ double yp = ray3d.at(0, 1);
+ double zp = ray3d.at(0, 2);
+ //inverse formula for spherical projection, reference Szeliski book "Computer Vision: Algorithms and Applications" p439.
+ double theta = atan2(yp, sqrt(xp * xp + zp * zp));
+ double phi = atan2(xp, zp);
+
+ //get 2D point on equirectangular map
+ double x_sphere = (((phi * img_src.cols) / pi + img_src.cols) / 2);
+ double y_sphere = (theta + pi / 2) * img_src.rows / pi;
+
+ return std::make_pair(x_sphere, y_sphere);
+}
+
+void rectPortionRaster(Raster420 & yuv_raster, cv::Mat& img_src) {
+
+ for ( unsigned int y = 0; y < SCREEN_RES_Y; y++ ) {
+ for ( unsigned int x = 0; x < SCREEN_RES_X; x++ ) {
+ //determine corresponding position in the equirectangular panorama
+ std::pair current_pos = reprojection(x, y, img_src);
+
+ //extract the x and y value of the position in the equirect. panorama
+ double current_x = current_pos.first;
+ double current_y = current_pos.second;
+
+ // if the current position exceeeds the panorama image limit -- leave pixel black and skip to next iteration
+ if (current_x < 0 || current_y < 0 )
+ {
+ continue;
+ }
+
+ cv::Vec3b bgr = img_src.at(current_y, current_x);
+
+ float blue = bgr.val[0]/255.0;
+ float green = bgr.val[1]/255.0;
+ float red = bgr.val[2]/255.0;
+
+ const float Ey = 0.7154 * green + 0.0721 * blue + 0.2125 * red;
+ const float Epb = -0.386 * green + 0.5000 * blue - 0.115 * red;
+ const float Epr = -0.454 * green - 0.046 * blue + 0.500 * red;
+
+ const uint8_t Y = (219 * Ey) + 16;
+ const uint8_t Cb = (224 * Epb) + 128;
+ const uint8_t Cr = (224 * Epr) + 128;
+
+ yuv_raster.Y.at( x, y ) = Y;
+ if ( (x%2) == 0 and (y%2) == 0 ) {
+ yuv_raster.Cb.at( x / 2, y / 2 ) = Cb;
+ yuv_raster.Cr.at( x / 2, y / 2 ) = Cr;
+ }
+ }
+ }
+ return;
+}
+
+void onlineMethod() {
+ cv::Mat img_src = cv::imread("/home/brooke/repos/eyelink-latency/src/files/frame92.png", cv::IMREAD_COLOR);
+ VideoDisplay display { SCREEN_RES_X, SCREEN_RES_Y, false }; // fullscreen window @ 1920x1080 luma resolution
+ Raster420 yuv_raster { SCREEN_RES_X , SCREEN_RES_Y };
+ rectPortionRaster(yuv_raster, img_src);
+ Texture420 texture { yuv_raster };
+
+ //std::cout << img_src.cols << std::endl;
+
+ while (true) {
+ display.draw( texture );
+ }
+}
+
+void withShader() {
+
+ VideoDisplay display { SCREEN_RES_X, SCREEN_RES_Y, false }; // fullscreen window @ 1920x1080 luma resolution
+ Raster420 yuv_raster { SCREEN_RES_X, SCREEN_RES_Y };
+ writePNGRaster(yuv_raster);
+ Texture420 texture { yuv_raster };
+
+ float roll = 0;
+ float pitch = 0;
+ float yaw = 0;
+
+ while (true) {
+ display.draw( texture );
+ display.update_head_orientation( roll, pitch, yaw );
+ roll += 0.001;
+ pitch += 0.003;
+ yaw += 0.001;
+ }
+
+}
+
+
+int main()
+{
+ try {
+ withShader();
+ //onlineMethod();
+
+ } catch ( const exception& e ) {
+ cerr << "Exception: " << e.what() << "\n";
+ return EXIT_FAILURE;
+ }
+
+ return EXIT_SUCCESS;
+}
diff --git a/src/frontend/example.cc b/src/frontend/example.cc
index f2afd06..3956da1 100644
--- a/src/frontend/example.cc
+++ b/src/frontend/example.cc
@@ -1,63 +1,336 @@
+#include
#include
#include
#include
+#include
#include
+#include
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
#include "display.hh"
+#define CURSOR_SIZE 5 /* radius of the white dot in px */
+#define NUM_TRIALS 1
+
using namespace std;
using namespace std::chrono;
-void program_body()
+/**
+ * End recording: adds 100 msec of data to catch final events
+ */
+static void end_trial( void )
{
- VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution
- display.window().hide_cursor( true );
- display.window().set_swap_interval( 1 ); // wait for vertical retrace before swapping buffer
+ pump_delay( 100 ); // provide a small amount of delay for last data
+ stop_recording();
+ while ( getkey() ) {
+ };
+}
+
+int get_tracker_sw_version( char* verstr )
+{
+ int ln = 0;
+ int st = 0;
+ ln = strlen( verstr );
+ while ( ln > 0 && verstr[ln - 1] == ' ' )
+ verstr[--ln] = 0; // trim
+
+ // find the start of the version number
+ st = ln;
+ while ( st > 0 && verstr[st - 1] != ' ' )
+ st--;
+ return atoi( &verstr[st] );
+}
+
+int initialize_eyelink()
+{
+ char verstr[50];
+ int eyelink_ver = 0;
+ int tracker_software_ver = 0;
+
+ // Set the address of the tracker, this is hard-coded by the Eyelink
+ set_eyelink_address( (char*)"100.1.1.1" );
+
+ // Initialize the EyeLink DLL and connect to the tracker
+ // * 0 opens a connection with the eye tracker
+ // * 1 will create a dummy connection for simulation
+ // * -1 initializes the DLL but does not open a connection
+ if ( open_eyelink_connection( 0 ) )
+ return -1;
- /* all white (235 = max luma in typical Y'CbCr colorspace) */
- Raster420 white { 1920, 1080 };
- memset( white.Y.mutable_pixels(), 235, white.Y.width() * white.Y.height() );
- Texture420 white_texture { white };
+ set_offline_mode();
+ flush_getkey_queue();
- /* left half white */
- Raster420 left_white { 1920, 1080 };
+ // Now configure tracker for display resolution
+ eyecmd_printf( "screen_pixel_coords = %ld %ld %ld %ld", 0, 0, 1920, 1080 );
- for ( unsigned int y = 0; y < left_white.Y.height(); y++ ) {
- for ( unsigned int x = 0; x < left_white.Y.width(); x++ ) {
- const uint8_t color = ( x < left_white.Y.width() / 2 ) ? 235 : 16;
- left_white.Y.at( x, y ) = color;
+ eyelink_ver = eyelink_get_tracker_version( verstr );
+ if ( eyelink_ver == 3 )
+ tracker_software_ver = get_tracker_sw_version( verstr );
+
+ // SET UP TRACKER CONFIGURATION
+ // set parser saccade thresholds (conservative settings)
+ if ( eyelink_ver >= 2 ) {
+ eyecmd_printf( "select_parser_configuration 0" ); // 0 = standard sensitivity
+ // turn off scenelink camera stuff
+ if ( eyelink_ver == 2 ) {
+ eyecmd_printf( "scene_camera_gazemap = NO" );
}
+ } else {
+ eyecmd_printf( "saccade_velocity_threshold = 35" );
+ eyecmd_printf( "saccade_acceleration_threshold = 9500" );
+ }
+
+ // set link data (used for gaze cursor)
+ eyecmd_printf( "link_event_filter = LEFT,RIGHT,FIXATION,SACCADE,BLINK,BUTTON,INPUT" );
+ eyecmd_printf( "link_sample_data = LEFT,RIGHT,GAZE,GAZERES,AREA,STATUS%s,INPUT",
+ ( tracker_software_ver >= 4 ) ? ",HTARGET" : "" );
+
+ // Make sure we're still alive
+ if ( !eyelink_is_connected() || break_pressed() ) {
+ return -1;
+ }
+ return 0;
+}
+
+int exit_eyelink()
+{
+ close_expt_graphics(); // tell EXPTSPPT to release window
+ close_eyelink_connection(); // disconnect from tracker
+ return 0;
+}
+
+void clear_full_screen_window( SDL_Surface* window, SDL_Color c )
+{
+ SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) );
+ SDL_Flip( window );
+ SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) );
+}
+
+void do_calibration()
+{
+ // The colors of the target and background for calibration and drift correct
+ SDL_Color target_background_color = { 192, 192, 192, 0 };
+ SDL_Color target_foreground_color = { 0, 0, 0, 0 };
+ SDL_Surface* window = NULL;
+
+ // register window with EXPTSPPT
+ if ( init_expt_graphics( NULL, NULL ) ) {
+ exit_eyelink();
+ return;
}
- Texture420 left_white_texture { left_white };
+ // Setup calibration type
+ eyecmd_printf( "calibration_type = HV9" );
- /* all black (16 = min luma in typical Y'CbCr colorspace) */
- Raster420 black { 1920, 1080 };
- memset( black.Y.mutable_pixels(), 16, black.Y.width() * black.Y.height() );
- Texture420 black_texture { black };
+ window = SDL_GetVideoSurface();
- /* alternate black and white */
+ // Size for calibration target and focal spot
+ unsigned int i = 1920 / 60;
+ unsigned int j = 1920 / 300;
+ if ( j < 2 )
+ j = 2;
+ set_target_size( i, j ); // tell DLL the size of target features
+
+ // tell EXPTSPPT the colors
+ set_calibration_colors( &target_foreground_color, &target_background_color );
+
+ clear_full_screen_window( window, target_background_color );
+
+ SDL_Flip( window );
+
+ do_tracker_setup();
+
+ // Wait for user to press escape before continuing
+ while ( !escape_pressed() ) {
+ };
+
+ close_expt_graphics(); // tell EXPTSPPT to release window
+}
+
+int gc_window_trial()
+{
+ ofstream log;
+ log.open( "results.csv" );
unsigned int frame_count = 0;
+ int button; /* the button pressed (0 if timeout) */
+
+ // First, set up all the textures
+ VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution
+ display.window().hide_cursor( true );
+
+ // whether to wait for vertical retrace before swapping buffer
+ // * 0 for immediate updates
+ // * 1 for updates synchronized with the vertical retrace
+ // * -1 for adaptive vsync
+ display.window().set_swap_interval( 0 );
+
+ // Used to track gaze samples
+ ALLF_DATA evt;
+ float x_sample, y_sample;
+
+ // Ensure Eyelink has enough time to switch modes
+ set_offline_mode();
+ pump_delay( 50 );
+
+ // Start data streaming
+ // Note that we are ignoring the EDF file.
+ int error = start_recording( 0, 0, 1, 1 );
+ if ( error != 0 ) {
+ log.close();
+ return error;
+ }
+
+ // wait for link sample data
+ if ( !eyelink_wait_for_block_start( 100, 1, 0 ) ) {
+ end_trial();
+ cerr << "ERROR: No link samples received!\n";
+ log.close();
+ return TRIAL_ERROR;
+ }
+
+ // determine which eye(s) are available
+ int eye_used = eyelink_eye_available();
+
+ // reset keys and buttons from tracker
+ eyelink_flush_keybuttons( 0 );
const auto start_time = steady_clock::now();
+ // Poll for new samples until the diff between samples is large enough to signify LEDs switched
while ( true ) {
- display.draw( left_white_texture );
- frame_count++;
- display.draw( white_texture );
- frame_count++;
- display.draw( black_texture );
- frame_count++;
+ // Termination conditions
+ if ( ( error = check_recording() ) != 0 ) {
+ log.close();
+ return error;
+ }
+
+ if ( break_pressed() ) /* check for program termination or ALT-F4 or CTRL-C keys */
+ {
+ end_trial(); /* local function to stop recording */
+ log.close();
+ return ABORT_EXPT; /* return this code to terminate experiment */
+ }
+
+ if ( escape_pressed() ) /* check for local ESC key to abort trial (useful in debugging) */
+ {
+ end_trial(); /* local function to stop recording */
+ log.close();
+ return SKIP_TRIAL; /* return this code if trial terminated */
+ }
+
+ /* BUTTON RESPONSE TEST */
+ /* Check for eye-tracker buttons pressed */
+ /* This is the preferred way to get response data or end trials */
+ button = eyelink_last_button_press( NULL );
+ if ( button != 0 ) /* button number, or 0 if none pressed */
+ {
+ end_trial(); /* local function to stop recording */
+ break; /* exit trial loop */
+ }
+
+ // check for new sample update
+ if ( eyelink_newest_float_sample( NULL ) > 0 ) {
+ eyelink_newest_float_sample( &evt );
+
+ x_sample = evt.fs.gx[eye_used];
+ y_sample = evt.fs.gy[eye_used];
- if ( frame_count % 480 == 0 ) {
- const auto now = steady_clock::now();
+ const auto t_sample = duration_cast( steady_clock::now() - start_time ).count();
+ log << t_sample << ", " << x_sample << ", " << y_sample << endl;
- const auto ms_elapsed = duration_cast( now - start_time ).count();
+ // make sure pupil is present
+ if ( x_sample != MISSING_DATA && y_sample != MISSING_DATA && evt.fs.pa[eye_used] > 0 ) {
+ // TODO: Draw a dot where we are looking. This is currently very naive,
+ // we should be able to draw a dot faster. Right now, this is limiting
+ // our FPS to about 77fps.
+
+ /* draw cursor location (235 = max luma in typical Y'CbCr colorspace) */
+ Raster420 cursor { 1920, 1080 };
- cout << "Drew " << frame_count << " frames in " << ms_elapsed
- << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n";
+ for ( unsigned int y = 0; y < cursor.Y.height(); y++ ) {
+ for ( unsigned int x = 0; x < cursor.Y.width(); x++ ) {
+ // Check if point is within radius
+ bool within_circle = ( ( ( x_sample - x ) * ( x_sample - x ) ) +
+ ( ( y_sample - y ) * ( y_sample - y ) ) ) <= ( CURSOR_SIZE * CURSOR_SIZE );
+ const uint8_t color = within_circle ? 235 : 16;
+ cursor.Y.at( x, y ) = color;
+ }
+ }
+
+ Texture420 cursor_texture { cursor };
+
+ // Draw texture. Note this may introduce extra delay since this is
+ // computing a texture each update.
+ display.draw( cursor_texture );
+
+ frame_count++;
+
+ if ( frame_count % 480 == 0 ) {
+ const auto now = steady_clock::now();
+ const auto ms_elapsed = duration_cast( now - start_time ).count();
+ cout << "Drew " << frame_count << " frames in " << ms_elapsed
+ << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n";
+ }
+ }
+ }
+ }
+ end_trial();
+ log.close();
+ return check_record_exit();
+}
+
+int run_trials()
+{
+
+ do_calibration();
+
+ for ( unsigned int trial = 0; trial < NUM_TRIALS; trial++ ) {
+ // abort if link is closed
+ if ( eyelink_is_connected() == 0 || break_pressed() ) {
+ return ABORT_EXPT;
}
+
+ int i = gc_window_trial();
+
+ // Report errors
+ switch ( i ) {
+ case ABORT_EXPT: // handle experiment abort or disconnect
+ cout << "EXPERIMENT ABORTED\n";
+ return ABORT_EXPT;
+ case REPEAT_TRIAL: // trial restart requested
+ cout << "TRIAL REPEATED\n";
+ trial--;
+ break;
+ case SKIP_TRIAL: // skip trial
+ cout << "TRIAL ABORTED\n";
+ break;
+ case TRIAL_OK: // successful trial
+ cout << "TRIAL OK\n";
+ break;
+ default: // other error code
+ cout << "TRIAL ERROR\n";
+ break;
+ }
+ }
+
+ return 0;
+}
+
+void program_body()
+{
+ if ( initialize_eyelink() < 0 ) {
+ cerr << "[Error] Unable to initialize EyeLink.\n";
+ exit( EXIT_FAILURE );
}
+ run_trials();
}
int main()
diff --git a/src/frontend/gazedraw.cc b/src/frontend/gazedraw.cc
new file mode 100644
index 0000000..966dca6
--- /dev/null
+++ b/src/frontend/gazedraw.cc
@@ -0,0 +1,171 @@
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+
+#include "cairo_objects.hh"
+#include "display.hh"
+
+using namespace std;
+using namespace std::chrono;
+
+void program_body()
+{
+ VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution
+
+ Cairo cairo { 1920, 1080 };
+ Pango pango { cairo };
+
+ /* open the PNG */
+ PNGSurface png_image { "/home/brooke/repos/eyelink-latency/src/files/testim.png" };
+
+ /* draw gray over everything */
+ cairo_new_path( cairo );
+ cairo_identity_matrix( cairo );
+ cairo_rectangle( cairo, 500, 500, 100, 100 );
+ cairo_set_source_rgba( cairo, 0, 0.9, 0, 0.5 );
+ cairo_fill( cairo );
+
+ /* draw the PNG */
+ cairo_identity_matrix( cairo );
+ cairo_scale( cairo, 0.8, 0.8 );
+ double center_x = 960, center_y = 540;
+ cairo_device_to_user( cairo, ¢er_x, ¢er_y );
+ cairo_translate( cairo, center_x, center_y );
+ cairo_set_source_surface( cairo, png_image, 0, 0 );
+ cairo_paint( cairo );
+
+ /* draw some text */
+ Pango::Font myfont { "Times New Roman, 80" };
+ Pango::Text mystring { cairo, pango, myfont, "Hello, world, Brooke, and Luke." };
+ mystring.draw_centered_at( cairo, 960, 540 );
+ cairo_set_source_rgba( cairo, 1, 1, 1, 1 );
+ cairo_fill( cairo );
+
+ /* finish and copy to YUV raster */
+ cairo.flush();
+
+ unsigned int stride = cairo.stride();
+ Raster420 yuv_raster { 1920, 1080 };
+ for ( unsigned int y = 0; y < 1080; y++ ) {
+ for ( unsigned int x = 0; x < 1920; x++ ) {
+ yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )];
+ }
+ }
+
+ Texture420 texture { yuv_raster };
+ display.draw( texture );
+ pause();
+}
+
+void writeTextRaster(Raster420 & yuv_raster, float pos_x, float pos_y) {
+ Cairo cairo { 1920, 1080 };
+ Pango pango { cairo };
+ cairo_new_path( cairo );
+
+ Pango::Font myfont { "Times New Roman, 80" };
+ Pango::Text mystring { cairo, pango, myfont, "Eye" };
+ mystring.draw_centered_at( cairo, pos_x * 1920, pos_y * 1080);
+ cairo_set_source_rgba( cairo, 1, 1, 1, 1 );
+ cairo_fill( cairo );
+ cairo.flush();
+
+ unsigned int stride = cairo.stride();
+ for ( unsigned int y = 0; y < 1080; y++ ) {
+ for ( unsigned int x = 0; x < 1920; x++ ) {
+ yuv_raster.Y.at( x, y ) = cairo.pixels()[y * stride + 1 + ( x * 4 )];
+ }
+ }
+
+}
+
+
+
+int gazeUpdate(float & pos_x)
+{
+ zmq::context_t context;
+
+ zmq::socket_t sock(context, zmq::socket_type::req);
+ fprintf(stderr, "Connecting to socket.\n");
+ sock.connect("tcp://127.0.0.1:4587");
+ fprintf(stderr, "Connected.\n");
+ //sock.send(zmq::str_buffer("R"), zmq::send_flags::dontwait);
+ // fprintf(stderr, "Sent.\n");
+
+ fprintf(stderr, "Send SUB_PORT.\n");
+ sock.send(zmq::str_buffer("SUB_PORT"), zmq::send_flags::dontwait);
+ zmq::message_t sub_port;
+ fprintf(stderr, "Recv SUB_PORT.\n");
+ auto ret = sock.recv(sub_port, zmq::recv_flags::none);
+ if (!ret)
+ return 1;
+ cout << "SUB_PORT: " << sub_port.to_string() << "\n";
+
+ sock.send(zmq::str_buffer("PUB_PORT"), zmq::send_flags::dontwait);
+ zmq::message_t pub_port;
+ ret = sock.recv(pub_port, zmq::recv_flags::none);
+ if (!ret)
+ return 1;
+ cout << "PUB_PORT: " << pub_port.to_string() << "\n";
+
+ zmq::socket_t subscriber(context, zmq::socket_type::sub);
+ subscriber.connect("tcp://127.0.0.1:" + sub_port.to_string());
+
+ subscriber.setsockopt(ZMQ_SUBSCRIBE, "gaze", strlen("gaze"));
+
+ while (true) {
+ vector recv_msgs;
+ ret = zmq::recv_multipart(subscriber, std::back_inserter(recv_msgs));
+ if (!ret)
+ return 1;
+
+ msgpack::object_handle oh = msgpack::unpack((const char*) recv_msgs[1].data(), recv_msgs[1].size());
+ msgpack::object obj = oh.get();
+ //cout << obj << endl;
+ //string eye = obj.via.array.ptr[1].as();
+ //cout << "eye: " << eye[8] << endl;
+ //float timestamp = obj.via.array.ptr[7].as();
+ //cout << "timestamp: " << timestamp << endl;
+ float pos1 = obj.via.array.ptr[3].via.array.ptr[0].as();
+ //float pos2 = obj.via.array.ptr[3].via.array.ptr[1].as();
+ //cout << "pos: " << "[" << pos1 << ", " << pos2 << "]" << endl;
+ if (!isnan(pos1)) {
+ pos_x = pos1;
+ }
+ }
+ return 0;
+}
+
+void test() {
+ VideoDisplay display { 1920, 1080, false }; // fullscreen window @ 1920x1080 luma resolution
+ Raster420 yuv_raster { 1920, 1080 };
+ float pos_x;
+ gazeUpdate(pos_x);
+ writeTextRaster(yuv_raster, pos_x, 0.5);
+ Texture420 texture { yuv_raster };
+ display.draw( texture );
+ pause();
+}
+
+int main()
+{
+ try {
+ //program_body();
+ test();
+ } catch ( const exception& e ) {
+ cerr << "Exception: " << e.what() << "\n";
+ return EXIT_FAILURE;
+ }
+
+ return EXIT_SUCCESS;
+}
diff --git a/src/frontend/pupil_labs.cc b/src/frontend/pupil_labs.cc
new file mode 100644
index 0000000..fd7a336
--- /dev/null
+++ b/src/frontend/pupil_labs.cc
@@ -0,0 +1,64 @@
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+
+using namespace std;
+
+int main()
+{
+ zmq::context_t context;
+
+ zmq::socket_t sock(context, zmq::socket_type::req);
+ fprintf(stderr, "Connecting to socket.\n");
+ sock.connect("tcp://127.0.0.1:4587");
+ fprintf(stderr, "Connected.\n");
+ //sock.send(zmq::str_buffer("R"), zmq::send_flags::dontwait);
+ // fprintf(stderr, "Sent.\n");
+
+ fprintf(stderr, "Send SUB_PORT.\n");
+ sock.send(zmq::str_buffer("SUB_PORT"), zmq::send_flags::dontwait);
+ zmq::message_t sub_port;
+ fprintf(stderr, "Recv SUB_PORT.\n");
+ auto ret = sock.recv(sub_port, zmq::recv_flags::none);
+ if (!ret)
+ return 1;
+ cout << "SUB_PORT: " << sub_port.to_string() << "\n";
+
+ sock.send(zmq::str_buffer("PUB_PORT"), zmq::send_flags::dontwait);
+ zmq::message_t pub_port;
+ ret = sock.recv(pub_port, zmq::recv_flags::none);
+ if (!ret)
+ return 1;
+ cout << "PUB_PORT: " << pub_port.to_string() << "\n";
+
+ zmq::socket_t subscriber(context, zmq::socket_type::sub);
+ subscriber.connect("tcp://127.0.0.1:" + sub_port.to_string());
+
+ subscriber.setsockopt(ZMQ_SUBSCRIBE, "gaze", strlen("gaze"));
+
+ while (true) {
+ vector recv_msgs;
+ ret = zmq::recv_multipart(subscriber, std::back_inserter(recv_msgs));
+ if (!ret)
+ return 1;
+
+ msgpack::object_handle oh = msgpack::unpack((const char*) recv_msgs[1].data(), recv_msgs[1].size());
+ msgpack::object obj = oh.get();
+ //cout << obj << endl;
+ string eye = obj.via.array.ptr[1].as();
+ cout << "eye: " << eye[8] << endl;
+ float timestamp = obj.via.array.ptr[7].as();
+ cout << "timestamp: " << timestamp << endl;
+ float pos1 = obj.via.array.ptr[3].via.array.ptr[0].as();
+ float pos2 = obj.via.array.ptr[3].via.array.ptr[1].as();
+ cout << "pos: " << "[" << pos1 << ", " << pos2 << "]" << endl;
+ }
+
+ return 0;
+}
diff --git a/src/frontend/user_test.cc b/src/frontend/user_test.cc
new file mode 100644
index 0000000..3956da1
--- /dev/null
+++ b/src/frontend/user_test.cc
@@ -0,0 +1,346 @@
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+
+#include "display.hh"
+
+#define CURSOR_SIZE 5 /* radius of the white dot in px */
+#define NUM_TRIALS 1
+
+using namespace std;
+using namespace std::chrono;
+
+/**
+ * End recording: adds 100 msec of data to catch final events
+ */
+static void end_trial( void )
+{
+ pump_delay( 100 ); // provide a small amount of delay for last data
+ stop_recording();
+ while ( getkey() ) {
+ };
+}
+
+int get_tracker_sw_version( char* verstr )
+{
+ int ln = 0;
+ int st = 0;
+ ln = strlen( verstr );
+ while ( ln > 0 && verstr[ln - 1] == ' ' )
+ verstr[--ln] = 0; // trim
+
+ // find the start of the version number
+ st = ln;
+ while ( st > 0 && verstr[st - 1] != ' ' )
+ st--;
+ return atoi( &verstr[st] );
+}
+
+int initialize_eyelink()
+{
+ char verstr[50];
+ int eyelink_ver = 0;
+ int tracker_software_ver = 0;
+
+ // Set the address of the tracker, this is hard-coded by the Eyelink
+ set_eyelink_address( (char*)"100.1.1.1" );
+
+ // Initialize the EyeLink DLL and connect to the tracker
+ // * 0 opens a connection with the eye tracker
+ // * 1 will create a dummy connection for simulation
+ // * -1 initializes the DLL but does not open a connection
+ if ( open_eyelink_connection( 0 ) )
+ return -1;
+
+ set_offline_mode();
+ flush_getkey_queue();
+
+ // Now configure tracker for display resolution
+ eyecmd_printf( "screen_pixel_coords = %ld %ld %ld %ld", 0, 0, 1920, 1080 );
+
+ eyelink_ver = eyelink_get_tracker_version( verstr );
+ if ( eyelink_ver == 3 )
+ tracker_software_ver = get_tracker_sw_version( verstr );
+
+ // SET UP TRACKER CONFIGURATION
+ // set parser saccade thresholds (conservative settings)
+ if ( eyelink_ver >= 2 ) {
+ eyecmd_printf( "select_parser_configuration 0" ); // 0 = standard sensitivity
+ // turn off scenelink camera stuff
+ if ( eyelink_ver == 2 ) {
+ eyecmd_printf( "scene_camera_gazemap = NO" );
+ }
+ } else {
+ eyecmd_printf( "saccade_velocity_threshold = 35" );
+ eyecmd_printf( "saccade_acceleration_threshold = 9500" );
+ }
+
+ // set link data (used for gaze cursor)
+ eyecmd_printf( "link_event_filter = LEFT,RIGHT,FIXATION,SACCADE,BLINK,BUTTON,INPUT" );
+ eyecmd_printf( "link_sample_data = LEFT,RIGHT,GAZE,GAZERES,AREA,STATUS%s,INPUT",
+ ( tracker_software_ver >= 4 ) ? ",HTARGET" : "" );
+
+ // Make sure we're still alive
+ if ( !eyelink_is_connected() || break_pressed() ) {
+ return -1;
+ }
+ return 0;
+}
+
+int exit_eyelink()
+{
+ close_expt_graphics(); // tell EXPTSPPT to release window
+ close_eyelink_connection(); // disconnect from tracker
+ return 0;
+}
+
+void clear_full_screen_window( SDL_Surface* window, SDL_Color c )
+{
+ SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) );
+ SDL_Flip( window );
+ SDL_FillRect( window, NULL, SDL_MapRGB( window->format, c.r, c.g, c.b ) );
+}
+
+void do_calibration()
+{
+ // The colors of the target and background for calibration and drift correct
+ SDL_Color target_background_color = { 192, 192, 192, 0 };
+ SDL_Color target_foreground_color = { 0, 0, 0, 0 };
+ SDL_Surface* window = NULL;
+
+ // register window with EXPTSPPT
+ if ( init_expt_graphics( NULL, NULL ) ) {
+ exit_eyelink();
+ return;
+ }
+
+ // Setup calibration type
+ eyecmd_printf( "calibration_type = HV9" );
+
+ window = SDL_GetVideoSurface();
+
+ // Size for calibration target and focal spot
+ unsigned int i = 1920 / 60;
+ unsigned int j = 1920 / 300;
+ if ( j < 2 )
+ j = 2;
+ set_target_size( i, j ); // tell DLL the size of target features
+
+ // tell EXPTSPPT the colors
+ set_calibration_colors( &target_foreground_color, &target_background_color );
+
+ clear_full_screen_window( window, target_background_color );
+
+ SDL_Flip( window );
+
+ do_tracker_setup();
+
+ // Wait for user to press escape before continuing
+ while ( !escape_pressed() ) {
+ };
+
+ close_expt_graphics(); // tell EXPTSPPT to release window
+}
+
+int gc_window_trial()
+{
+ ofstream log;
+ log.open( "results.csv" );
+ unsigned int frame_count = 0;
+ int button; /* the button pressed (0 if timeout) */
+
+ // First, set up all the textures
+ VideoDisplay display { 1920, 1080, true }; // fullscreen window @ 1920x1080 luma resolution
+ display.window().hide_cursor( true );
+
+ // whether to wait for vertical retrace before swapping buffer
+ // * 0 for immediate updates
+ // * 1 for updates synchronized with the vertical retrace
+ // * -1 for adaptive vsync
+ display.window().set_swap_interval( 0 );
+
+ // Used to track gaze samples
+ ALLF_DATA evt;
+ float x_sample, y_sample;
+
+ // Ensure Eyelink has enough time to switch modes
+ set_offline_mode();
+ pump_delay( 50 );
+
+ // Start data streaming
+ // Note that we are ignoring the EDF file.
+ int error = start_recording( 0, 0, 1, 1 );
+ if ( error != 0 ) {
+ log.close();
+ return error;
+ }
+
+ // wait for link sample data
+ if ( !eyelink_wait_for_block_start( 100, 1, 0 ) ) {
+ end_trial();
+ cerr << "ERROR: No link samples received!\n";
+ log.close();
+ return TRIAL_ERROR;
+ }
+
+ // determine which eye(s) are available
+ int eye_used = eyelink_eye_available();
+
+ // reset keys and buttons from tracker
+ eyelink_flush_keybuttons( 0 );
+
+ const auto start_time = steady_clock::now();
+
+ // Poll for new samples until the diff between samples is large enough to signify LEDs switched
+ while ( true ) {
+ // Termination conditions
+ if ( ( error = check_recording() ) != 0 ) {
+ log.close();
+ return error;
+ }
+
+ if ( break_pressed() ) /* check for program termination or ALT-F4 or CTRL-C keys */
+ {
+ end_trial(); /* local function to stop recording */
+ log.close();
+ return ABORT_EXPT; /* return this code to terminate experiment */
+ }
+
+ if ( escape_pressed() ) /* check for local ESC key to abort trial (useful in debugging) */
+ {
+ end_trial(); /* local function to stop recording */
+ log.close();
+ return SKIP_TRIAL; /* return this code if trial terminated */
+ }
+
+ /* BUTTON RESPONSE TEST */
+ /* Check for eye-tracker buttons pressed */
+ /* This is the preferred way to get response data or end trials */
+ button = eyelink_last_button_press( NULL );
+ if ( button != 0 ) /* button number, or 0 if none pressed */
+ {
+ end_trial(); /* local function to stop recording */
+ break; /* exit trial loop */
+ }
+
+ // check for new sample update
+ if ( eyelink_newest_float_sample( NULL ) > 0 ) {
+ eyelink_newest_float_sample( &evt );
+
+ x_sample = evt.fs.gx[eye_used];
+ y_sample = evt.fs.gy[eye_used];
+
+ const auto t_sample = duration_cast( steady_clock::now() - start_time ).count();
+ log << t_sample << ", " << x_sample << ", " << y_sample << endl;
+
+ // make sure pupil is present
+ if ( x_sample != MISSING_DATA && y_sample != MISSING_DATA && evt.fs.pa[eye_used] > 0 ) {
+ // TODO: Draw a dot where we are looking. This is currently very naive,
+ // we should be able to draw a dot faster. Right now, this is limiting
+ // our FPS to about 77fps.
+
+ /* draw cursor location (235 = max luma in typical Y'CbCr colorspace) */
+ Raster420 cursor { 1920, 1080 };
+
+ for ( unsigned int y = 0; y < cursor.Y.height(); y++ ) {
+ for ( unsigned int x = 0; x < cursor.Y.width(); x++ ) {
+ // Check if point is within radius
+ bool within_circle = ( ( ( x_sample - x ) * ( x_sample - x ) ) +
+ ( ( y_sample - y ) * ( y_sample - y ) ) ) <= ( CURSOR_SIZE * CURSOR_SIZE );
+ const uint8_t color = within_circle ? 235 : 16;
+ cursor.Y.at( x, y ) = color;
+ }
+ }
+
+ Texture420 cursor_texture { cursor };
+
+ // Draw texture. Note this may introduce extra delay since this is
+ // computing a texture each update.
+ display.draw( cursor_texture );
+
+ frame_count++;
+
+ if ( frame_count % 480 == 0 ) {
+ const auto now = steady_clock::now();
+ const auto ms_elapsed = duration_cast( now - start_time ).count();
+ cout << "Drew " << frame_count << " frames in " << ms_elapsed
+ << " milliseconds = " << 1000.0 * double( frame_count ) / ms_elapsed << " frames per second.\n";
+ }
+ }
+ }
+ }
+ end_trial();
+ log.close();
+ return check_record_exit();
+}
+
+int run_trials()
+{
+
+ do_calibration();
+
+ for ( unsigned int trial = 0; trial < NUM_TRIALS; trial++ ) {
+ // abort if link is closed
+ if ( eyelink_is_connected() == 0 || break_pressed() ) {
+ return ABORT_EXPT;
+ }
+
+ int i = gc_window_trial();
+
+ // Report errors
+ switch ( i ) {
+ case ABORT_EXPT: // handle experiment abort or disconnect
+ cout << "EXPERIMENT ABORTED\n";
+ return ABORT_EXPT;
+ case REPEAT_TRIAL: // trial restart requested
+ cout << "TRIAL REPEATED\n";
+ trial--;
+ break;
+ case SKIP_TRIAL: // skip trial
+ cout << "TRIAL ABORTED\n";
+ break;
+ case TRIAL_OK: // successful trial
+ cout << "TRIAL OK\n";
+ break;
+ default: // other error code
+ cout << "TRIAL ERROR\n";
+ break;
+ }
+ }
+
+ return 0;
+}
+
+void program_body()
+{
+ if ( initialize_eyelink() < 0 ) {
+ cerr << "[Error] Unable to initialize EyeLink.\n";
+ exit( EXIT_FAILURE );
+ }
+ run_trials();
+}
+
+int main()
+{
+ try {
+ program_body();
+ } catch ( const exception& e ) {
+ cerr << "Exception: " << e.what() << "\n";
+ return EXIT_FAILURE;
+ }
+
+ return EXIT_SUCCESS;
+}
diff --git a/src/util/Makefile.am b/src/util/Makefile.am
index c010a78..977d008 100644
--- a/src/util/Makefile.am
+++ b/src/util/Makefile.am
@@ -1,6 +1,7 @@
-AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLFW3_CFLAGS) $(GLEW_CFLAGS)
+AM_CPPFLAGS = $(CXX17_FLAGS) $(GLU_CFLAGS) $(GLFW3_CFLAGS) $(GLEW_CFLAGS) $(PANGOCAIRO_CFLAGS)
AM_CXXFLAGS = $(PICKY_CXXFLAGS)
noinst_LIBRARIES = libgldemoutil.a
-libgldemoutil_a_SOURCES = gl_objects.hh gl_objects.cc display.hh display.cc
+libgldemoutil_a_SOURCES = gl_objects.hh gl_objects.cc display.hh display.cc \
+ cairo_objects.hh cairo_objects.cc
diff --git a/src/util/cairo_objects.cc b/src/util/cairo_objects.cc
new file mode 100644
index 0000000..1288ea8
--- /dev/null
+++ b/src/util/cairo_objects.cc
@@ -0,0 +1,181 @@
+#include "cairo_objects.hh"
+
+#include
+#include
+
+using namespace std;
+
+Surface::Surface( cairo_surface_t* surface_ptr )
+ : surface_( surface_ptr )
+{
+ if ( not surface_ptr ) {
+ throw runtime_error( "surface_ptr is null" );
+ }
+
+ check_error();
+}
+
+void Surface::Deleter::operator()( cairo_surface_t* x ) const
+{
+ cairo_surface_destroy( x );
+}
+
+void Cairo::Context::Deleter::operator()( cairo_t* x ) const
+{
+ cairo_destroy( x );
+}
+
+void Cairo::Pattern::Deleter::operator()( cairo_pattern_t* x ) const
+{
+ cairo_pattern_destroy( x );
+}
+
+void Pango::Font::Deleter::operator()( PangoFontDescription* x ) const
+{
+ pango_font_description_free( x );
+}
+
+void Pango::Text::Deleter::operator()( cairo_path_t* x ) const
+{
+ cairo_path_destroy( x );
+}
+
+Cairo::Cairo( const unsigned int width, const unsigned int height )
+ : surface_( width, height )
+ , context_( surface_ )
+{
+ check_error();
+}
+
+ImageSurface::ImageSurface( cairo_surface_t* surface_ptr )
+ : Surface( surface_ptr )
+ , width_( cairo_image_surface_get_width( *this ) )
+ , height_( cairo_image_surface_get_height( *this ) )
+ , stride_( cairo_image_surface_get_stride( *this ) )
+{
+ check_error();
+}
+
+Cairo::Context::Context( ImageSurface& surface )
+ : context( cairo_create( surface ) )
+{
+ check_error();
+}
+
+void Surface::check_error()
+{
+ const cairo_status_t surface_result = cairo_surface_status( surface_.get() );
+ if ( surface_result ) {
+ throw runtime_error( string( "cairo surface error: " ) + cairo_status_to_string( surface_result ) );
+ }
+}
+
+void Cairo::Context::check_error()
+{
+ const cairo_status_t context_result = cairo_status( context.get() );
+ if ( context_result ) {
+ throw runtime_error( string( "cairo context error: " ) + cairo_status_to_string( context_result ) );
+ }
+}
+
+void Cairo::check_error()
+{
+ context_.check_error();
+ surface_.check_error();
+}
+
+Pango::Pango( Cairo& cairo )
+ : context_( pango_cairo_create_context( cairo ) )
+ , layout_( pango_layout_new( *this ) )
+{}
+
+Pango::Font::Font( const string& description )
+ : font( pango_font_description_from_string( description.c_str() ) )
+{}
+
+void Pango::set_font( const Pango::Font& font )
+{
+ pango_layout_set_font_description( *this, font );
+}
+
+mutex& global_pango_mutex()
+{
+ static mutex global_pango_mutex_;
+
+ return global_pango_mutex_;
+}
+
+Pango::Text::Text( Cairo& cairo, Pango& pango, const Font& font, const string& text )
+ : path_()
+ , extent_( { 0, 0, 0, 0 } )
+{
+ unique_lock ul { global_pango_mutex() };
+
+ cairo_identity_matrix( cairo );
+ cairo_new_path( cairo );
+
+ pango.set_font( font );
+
+ pango_layout_set_markup( pango, text.data(), text.size() );
+
+ pango_cairo_layout_path( cairo, pango );
+
+ path_.reset( cairo_copy_path( cairo ) );
+
+ /* get logical extents */
+ PangoRectangle logical;
+ pango_layout_get_extents( pango, nullptr, &logical );
+ extent_ = { logical.x / double( PANGO_SCALE ),
+ logical.y / double( PANGO_SCALE ),
+ logical.width / double( PANGO_SCALE ),
+ logical.height / double( PANGO_SCALE ) };
+}
+
+void Pango::Text::draw_centered_at( Cairo& cairo, const double x, const double y, const double max_width ) const
+{
+ cairo_identity_matrix( cairo );
+ cairo_new_path( cairo );
+ Cairo::Extent my_extent = extent().to_device( cairo );
+
+ double center_x = x - my_extent.x - my_extent.width / 2;
+ double center_y = y - my_extent.y - my_extent.height / 2;
+
+ if ( my_extent.width > max_width ) {
+ const double scale_factor = max_width / my_extent.width;
+ cairo_scale( cairo, scale_factor, scale_factor );
+ center_x = x - my_extent.x - my_extent.width * scale_factor / 2;
+ center_y = y - my_extent.y - my_extent.height * scale_factor / 2;
+ }
+
+ cairo_device_to_user( cairo, ¢er_x, ¢er_y );
+
+ cairo_translate( cairo, center_x, center_y );
+
+ cairo_append_path( cairo, path_.get() );
+}
+
+void Pango::Text::draw_centered_rotated_at( Cairo& cairo, const double x, const double y ) const
+{
+ cairo_identity_matrix( cairo );
+ cairo_new_path( cairo );
+
+ cairo_rotate( cairo, -3.1415926 / 2.0 );
+
+ Cairo::Extent my_extent = extent().to_device( cairo );
+
+ double center_x = x - my_extent.x - my_extent.width / 2;
+ double center_y = y - my_extent.y - my_extent.height / 2;
+
+ cairo_device_to_user( cairo, ¢er_x, ¢er_y );
+ cairo_translate( cairo, center_x, center_y );
+ cairo_append_path( cairo, path_.get() );
+}
+
+Cairo::Pattern::Pattern( cairo_pattern_t* pattern )
+ : pattern_( pattern )
+{
+ const cairo_status_t pattern_result = cairo_pattern_status( pattern_.get() );
+ if ( pattern_result ) {
+ throw runtime_error( string( "cairo pattern error: " ) + cairo_status_to_string( pattern_result ) );
+ }
+}
diff --git a/src/util/cairo_objects.hh b/src/util/cairo_objects.hh
new file mode 100644
index 0000000..68c94a7
--- /dev/null
+++ b/src/util/cairo_objects.hh
@@ -0,0 +1,198 @@
+#ifndef CAIRO_OBJECTS_HH
+#define CAIRO_OBJECTS_HH
+
+#include
+#include
+#include
+#include
+
+#include "gl_objects.hh"
+
+class Surface
+{
+ struct Deleter
+ {
+ void operator()( cairo_surface_t* x ) const;
+ };
+
+ std::unique_ptr surface_;
+
+protected:
+ Surface( cairo_surface_t* surface_ptr );
+
+public:
+ void check_error();
+
+ operator cairo_surface_t*() { return surface_.get(); }
+ operator const cairo_surface_t*() const { return surface_.get(); }
+};
+
+class ImageSurface : public Surface
+{
+ unsigned int width_, height_, stride_;
+
+protected:
+ ImageSurface( cairo_surface_t* surface_ptr );
+
+public:
+ uint8_t* pixels() { return cairo_image_surface_get_data( *this ); }
+
+ unsigned int width() const { return width_; }
+ unsigned int height() const { return height_; }
+ unsigned int stride() const { return stride_; }
+};
+
+class FreshImageSurface : public ImageSurface
+{
+public:
+ FreshImageSurface( const unsigned int width, const unsigned int height )
+ : ImageSurface( cairo_image_surface_create( CAIRO_FORMAT_RGB24, width, height ) )
+ {}
+};
+
+class PNGSurface : public ImageSurface
+{
+public:
+ PNGSurface( const char* filename )
+ : ImageSurface( cairo_image_surface_create_from_png( filename ) )
+ {}
+};
+
+class Cairo
+{
+ FreshImageSurface surface_;
+
+ struct Context
+ {
+ struct Deleter
+ {
+ void operator()( cairo_t* x ) const;
+ };
+
+ std::unique_ptr context;
+
+ Context( ImageSurface& surface );
+
+ void check_error();
+ } context_;
+
+ void check_error();
+
+public:
+ Cairo( const unsigned int width, const unsigned int height );
+
+ unsigned int width() { return surface_.width(); }
+ unsigned int height() { return surface_.height(); }
+ unsigned int stride() { return surface_.stride(); }
+
+ operator cairo_t*() { return context_.context.get(); }
+
+ uint8_t* pixels() { return surface_.pixels(); }
+ void flush() { cairo_surface_flush( surface_ ); }
+
+ template
+ struct Extent
+ {
+ double x, y, width, height;
+
+ Extent to_user( Cairo& cairo ) const
+ {
+ static_assert( device_coordinates == true, "Extent::to_user() called but coordinates already in user-space" );
+
+ double x1 = x, x2 = x + width, y1 = y, y2 = y + height;
+
+ cairo_device_to_user( cairo, &x1, &y1 );
+ cairo_device_to_user( cairo, &x2, &y2 );
+
+ return Extent( { x1, y1, x2 - x1, y2 - y1 } );
+ }
+
+ Extent to_device( Cairo& cairo ) const
+ {
+ static_assert( device_coordinates == false,
+ "Extent::to_device() called but coordinates already in device-space" );
+
+ double x1 = x, x2 = x + width, y1 = y, y2 = y + height;
+
+ cairo_user_to_device( cairo, &x1, &y1 );
+ cairo_user_to_device( cairo, &x2, &y2 );
+
+ return Extent( { x1, y1, x2 - x1, y2 - y1 } );
+ }
+ };
+
+ class Pattern
+ {
+ struct Deleter
+ {
+ void operator()( cairo_pattern_t* x ) const;
+ };
+
+ std::unique_ptr pattern_;
+
+ public:
+ Pattern( cairo_pattern_t* pattern );
+
+ operator cairo_pattern_t*() { return pattern_.get(); }
+ };
+};
+
+template
+struct PangoDelete
+{
+ void operator()( T* x ) const { g_object_unref( x ); }
+};
+
+class Pango
+{
+ std::unique_ptr> context_;
+ std::unique_ptr> layout_;
+
+public:
+ Pango( Cairo& cairo );
+
+ operator PangoContext*() { return context_.get(); }
+ operator PangoLayout*() { return layout_.get(); }
+
+ struct Font
+ {
+ struct Deleter
+ {
+ void operator()( PangoFontDescription* x ) const;
+ };
+
+ std::unique_ptr font;
+
+ Font( const std::string& description );
+
+ operator const PangoFontDescription*() const { return font.get(); }
+ };
+
+ class Text
+ {
+ struct Deleter
+ {
+ void operator()( cairo_path_t* x ) const;
+ };
+
+ std::unique_ptr path_;
+ Cairo::Extent extent_;
+
+ public:
+ Text( Cairo& cairo, Pango& pango, const Font& font, const std::string& text );
+
+ const Cairo::Extent& extent() const { return extent_; }
+
+ void draw_centered_at( Cairo& cairo,
+ const double x,
+ const double y,
+ const double max_width = std::numeric_limits::max() ) const;
+ void draw_centered_rotated_at( Cairo& cairo, const double x, const double y ) const;
+
+ operator const cairo_path_t*() const { return path_.get(); }
+ };
+
+ void set_font( const Font& font );
+};
+
+#endif /* CAIRO_OBJECTS_HH */
diff --git a/src/util/display.cc b/src/util/display.cc
index b3dd897..8da6976 100644
--- a/src/util/display.cc
+++ b/src/util/display.cc
@@ -36,14 +36,14 @@ const string VideoDisplay::shader_source_scale_from_pixel_coordinates = R"( #ver
in vec2 position;
in vec2 chroma_texcoord;
- out vec2 raw_position;
+ out vec2 Y_texcoord;
out vec2 uv_texcoord;
void main()
{
gl_Position = vec4( 2 * position.x / window_size.x - 1.0,
1.0 - 2 * position.y / window_size.y, 0.0, 1.0 );
- raw_position = vec2( position.x, position.y );
+ Y_texcoord = vec2( position.x, position.y );
uv_texcoord = vec2( chroma_texcoord.x, chroma_texcoord.y );
}
)";
@@ -70,15 +70,59 @@ const string VideoDisplay::shader_source_ycbcr = R"( #version 130
uniform sampler2DRect uTex;
uniform sampler2DRect vTex;
+ uniform vec3 head_orientation;
+
+ in vec2 Y_texcoord;
in vec2 uv_texcoord;
- in vec2 raw_position;
- out vec4 outColor;
+ out vec4 outColor;
+
+ mat3 eul2rotm( float rotX, float rotY, float rotZ ) {
+
+ mat3 R_x = mat3( 1.0f, 0f, 0f,
+ 0f, cos(rotX), sin(rotX),
+ 0f, -sin(rotX), cos(rotX));
+ mat3 R_y = mat3( cos(rotY), 0f, -sin(rotY),
+ 0f, 1.0f, 0f,
+ sin(rotY), 0f, cos(rotY));
+ mat3 R_z = mat3( cos(rotZ), sin(rotZ), 0f,
+ -sin(rotZ), cos(rotZ), 0f,
+ 0f, 0f, 1.0f);
+ mat3 invCamMat = mat3(0.0015f, 0f, 0f,
+ 0f, 0.0015f, 0f,
+ -1.4286f, -0.8036f, 1f);
+
+ return R_z * R_y * R_x * invCamMat;
+ }
+
+ vec2 get_latlong( vec2 Y_texcoord ) {
+ vec3 xyz = vec3( Y_texcoord.x, Y_texcoord.y, 1.0 );
+ vec3 xyz_norm = xyz / length(xyz);
+
+ vec3 ray3d = eul2rotm(head_orientation.x, head_orientation.y, head_orientation.z) * xyz_norm;
+ float theta = atan( ray3d.y, length(ray3d.xz) );
+ float phi = atan( ray3d.x, ray3d.z );
+ return vec2( phi, theta);
+ }
+
+ vec2 reproject_Y (vec2 Y_texcoord ) {
+ vec2 phi_theta = get_latlong(Y_texcoord);
+ vec2 xy_sphere = vec2( ((phi_theta.x / 3.14f) * 1920f + 1920.0)/2.0, (phi_theta.y + 3.14/2.0) * 1080f /3.14 );
+ return xy_sphere;
+ }
+
+ vec2 reproject_uv( vec2 Y_texcoord ) {
+ vec2 phi_theta = get_latlong(Y_texcoord);
+ vec2 xy_sphere = vec2( ((phi_theta.x / 3.14f) * 960f + 960.0)/2.0, (phi_theta.y + 3.14/2.0) * 540f /3.14 );
+ return xy_sphere;
+ }
void main()
{
- float fY = texture(yTex, raw_position).x;
- float fCb = texture(uTex, uv_texcoord).x;
- float fCr = texture(vTex, uv_texcoord).x;
+ vec2 Y_reprojected = reproject_Y(Y_texcoord);
+ vec2 uv_reprojected = reproject_uv(Y_texcoord) + max( 0.0, min( 0.0, texture(uTex, uv_texcoord).x ) ); // to get rid of inefficiency bug
+ float fY = texture(yTex, Y_reprojected).x;
+ float fCb = texture(uTex, uv_reprojected).x;
+ float fCr = texture(vTex, uv_reprojected).x;
outColor = vec4(
max(0, min(1.0, 1.16438356164384 * (fY - 0.06274509803921568627) + 1.59567019581339 * (fCr - 0.50196078431372549019))),
@@ -128,6 +172,12 @@ VideoDisplay::VideoDisplay( const unsigned int width, const unsigned int height,
glCheck( "VideoDisplay constructor" );
}
+void VideoDisplay::update_head_orientation( const float x, const float y, const float z )
+{
+ texture_shader_program_.use();
+ glUniform3f( texture_shader_program_.uniform_location( "head_orientation" ), x, y, z );
+}
+
void VideoDisplay::resize( const unsigned int width, const unsigned int height )
{
glViewport( 0, 0, width, height );
@@ -167,6 +217,12 @@ void VideoDisplay::resize( const unsigned int width, const unsigned int height )
glCheck( "after installing shaders" );
}
+// void VideoDisplay::updateHeadOrientation( const float pitch, const float roll, const float yaw) {
+// texture_shader_program_.use();
+// glUniform3f( texture_shader_program_.uniform_location( "head_orientation" ), pitch, roll, yaw );
+// }
+
+
void VideoDisplay::draw( Texture420& image )
{
image.bind();
diff --git a/src/util/display.hh b/src/util/display.hh
index e3aab0a..561ea8f 100644
--- a/src/util/display.hh
+++ b/src/util/display.hh
@@ -68,9 +68,13 @@ public:
void repaint();
void resize( const unsigned int width, const unsigned int height );
+ //void updateHeadOrientation( const float pitch, const float roll, const float yaw);
+
Window& window() { return current_context_window_.window_; }
const Window& window() const { return current_context_window_.window_; }
+ void update_head_orientation( const float x, const float y, const float z );
+
/* forbid copying */
VideoDisplay( const VideoDisplay& other ) = delete;
VideoDisplay& operator=( const VideoDisplay& other ) = delete;