updated readme

Author: jhdewitt

README.md

@@ -1,40 +1,85 @@
+# SLTK
+
+SLTK stands for Structured Light Took Kit. These are a collection of open source command line and GUI tools I have written with the intention of making operation of DIY structured light systems easier for myself. An effort has been made to separate the tools into logical units of work. It is possible to run the display program, the camera capture program, and the control program all on separate networked computers.
+
+Here's an example of a sea shell generated from 74 aligned pointclouds captured using a raspberry pi v1 camera + sony mp-cl1 projector + arduino-based turntable (~5deg per turn): [https://sketchfab.com/models/5def256988594fae93ceb69b59fabbcc](https://sketchfab.com/models/5def256988594fae93ceb69b59fabbcc)
+
+Two sets of raw+processed data from individual sea shell scans are available in the sampledata directory, along with the associated projector/camera calibration file.
+
+## Uses
+
+* Finding **intrinsic camera calibration**: 3x3 matrix + radial/tangential distortion coefficients
+* Finding **intrinsic projector calibration**: 3x3 matrix + radial/tangential distortion coefficients
+* Finding **extrinsic matrix** for a stereo camera + projector setup
+* Calculating **correspondence map** between camera and display pixels (screen or projector)
+* Generating **3D point clouds** from **correspondence data** and **intrinsic+extrinsic calibration data**
+* Rendering a **visualization of calibration data** with rainbow color map and isolines of distortion magnitude per pixel
+* **Displaying** timed structured light patterns via graphics adapter to display or projector
+* **Capturing** timed images of structured light patterns via camera input
+* **Automated rotation** of objects using a basic arduino stepper motor turntable
+* **Merging multiple point clouds** for generating a static background point set to allow automated cleanup
+* **Background subtraction** for cleaning up 3D point clouds by specifying a "background" point set
+* **Automated alignment** of 3D point clouds (requires background subtraction to remove static objects)
+* **Cultural Heritage Preservation**
+
 ## Synopsis
 
-The main goal of structured light scanning is to calculate the dense correspondence between camera pixels and output pixels. By displaying black and white patterns to signal 0 and 1, a flat panel display or projector can simply "speak in binary" the pixel coordinates to the camera. Multiple patterns are shown to signal each bit of the per-pixel address code (e.g. x output pixel address equals 10 -> binary 1010 so white-black-white-black is displayed at that output device pixel). The process of decoding images of displayed black-white patterns is executed as a simple rule for all camera pixels (was pixel bright or not).
+This collection of programs is intended to be used to create dense 3D pointclouds of semi-matte objects.
+This toolkit requires hardware in order to scan objects. Hardware tried for development includes:
+
+1) raspberry pi (v3 model b and zero)
+2) rpi camera (v1 and v2)
+3) flatscreen LCD monitor
+4) projector
+5) turntable
 
-Binary gray code structured light scanning involves displaying, capturing, and parsing sequences of black and white patterns. Uses include camera lens calibration and 3D point cloud reconstruction. By displaying specific patterns (on a display or projector) and capturing images of them, it is possible to calculate the correspondence between camera pixels and outut device pixels. More details available at https://en.wikipedia.org/wiki/Correspondence_problem https://en.wikipedia.org/wiki/Structured_Light_3D_Scanner
+## mjpeg-streamer [external tool]
+
+This third party program (https://github.com/jacksonliam/mjpg-streamer) continuously captures images from a camera and streams them over the network. A fork is used in this case to specifically ensure support for the rpi camera, although it should be possible to use other cameras. Please see mjpg-streamer docs for more info.
+Requires: camera
 
 ## sldisp
 
-Tool for displaying binary gray code patterns using an output device (e.g. LCD screen, DLP projector). Receives pattern change instructions over network (default localhost). Pattern input is OSC packets /sl/iii (int,int,int) for (bit,axis,inverted) 0=false, 1=true
+Tool for generating structured light patterns. Supports binary gray code, sinusoid, and monochrome RGB pattern outputs. Receives OSC messages and updates output pattern.
+Requires: display device (e.g. LCD screen, DLP projector), slcapture
+
+## slcapture.py
+
+Main control program: streams images continuously from an mjpg-streamer server while triggering preset pattern sequences on an sldisp client.
+Each "scan" output is really just a directory of images, each taken while a specific structured light pattern is displayed. These scans are processed by slcrunch.
+Requires: mjpg-streamer, camera, sldisp, display device
 
 ## slcrunch
 
-A tool for processing image sequences of binary gray code patterns into correspondence data or 3D point clouds. Takes a list of images as input. See listcreator for format of input file list.
+A tool for processing image sequences of e.g. binary gray code patterns into raw correspondence data or 3D point clouds. Requires *accurate* projector-camera calibration file in order to generate useful point clouds. Takes a list of images as input. See sampledata directory for list format.
 
 ## slcalibrate
 
-A tool for camera and projector lens calibration, as well as camera-projector registration. For camera lens calibration, a flatscreen display is used to show patterns and only camera~display correspondence data is required. For projector-camera calibration, a camera, projector, and a flat printed chessboard pattern is needed (see slcrunch -help for details).
+A tool for camera and projector lens calibration, as well as camera-projector registration. For camera lens calibration, a flatscreen display is used to show binary gray code patterns and only camera~display correspondence data is required. For projector-camera calibration, a camera, projector, and a flat printed chessboard pattern is needed (see slcrunch -help for details). See wiki for more info.
 
 ## chessgen
 
 A tool for generating images of chess patterns for lens calibration. Specify edge length in pixels, number of x/y inner corners, and x/y image output dimensions.
 
 ## plotlens
 
-This program creates an image visualizing image distortion using calibration data input. Helpful for understanding calibration data.
+This program creates an image visualizing image distortion using calibration data input. Helpful for understanding quality of calibration data.
 
 ## plyalign
 
-(WIP) Tool for automatically aligning multiply .PLY 3D pointcloud files. Specify .PLY files in sequential order, and repeated rotation from scan to scan is estimated.
+(WIP) Tool for automatically aligning multiple .PLY 3D pointcloud files. Specify .PLY files in sequential order, and repeated rotation from scan to scan is estimated.
+
+## plymerge
+
+Tool for merging multiple .PLY 3D pointcloud files. Combines all input files into one as-is with no alignment performed. Handy for making background subtraction files for speedy cleanup.
 
 ## plytrim
 
-(WIP) Tool for trimming .PLY 3D pointcloud files. Specify center coordinates and dimensions of 3D axis-aligned bounding box.
+Tool for trimming .PLY 3D pointcloud files. Input a primary/reference pointcloud + distance X: all points in primary set within X units of any point in reference are removed. Output file only contains points in primary set which have no neighbors within X units in reference set.
 
 ## listcreator
 
-This program creates a file containing a list of images in a sequence. Since image sequences need to be processed as a whole, image filename lists are used to pass them to slcrunch and slcalibrate. Capture programs should generate lists automatically, as in guppy_cap.cpp.
+This program creates a file containing a list of images in a sequence. Since image sequences need to be processed as a group, image filename lists are used to pass them to slcrunch and slcalibrate. Capture programs should generate lists automatically, as in slcapture.py. This tool is a manual fallback.
 
 ## Prerequisites
 
@@ -45,16 +90,23 @@ opencv2 pcl2 liblo sdl2 flann boost
 
 Built using: OpenCV(2.4.10), SDL2(2.0.3), liblo(0.28), exiftool(9.69), on Mac OS X 10.10.2, 10.11.6, 10.12
 
-Some programs link with OpenCV, an open source image processing library <http://opencv.org/>.
+mjpg-streamer is used to ingest camera data <https://github.com/jacksonliam/mjpg-streamer>
+Some programs link with OpenCV, an open source image processing library <http://opencv.org/>
 slcrunch calls the exiftool program for reading image metadata. <http://www.sno.phy.queensu.ca/~phil/exiftool/>
-sldisp links with SDL2, a user interface library. More information at <https://www.libsdl.org/>.
+sldisp links with SDL2, a user interface library. More information at <https://www.libsdl.org/>
 Some programs with liblo, an Open Sound Control library for packet communication. <http://opensoundcontrol.org/implementations>
 
-I have a basic Makefile; it should compile all the tools with "make" or individual ones with a target e.g. "make slcrunch".
+I have prepared a basic Makefile; it should compile all the tools with "make" or individual ones with a target e.g. "make slcrunch". Could use some work.
+
+## Separation of tasks
 
-## Use
+sltk is split up into standalone command line programs, each meant to handle a separate set of tasks in a structured light processing workflow.
 
-sltk is comprised of various standalone command-line programs to handle separate tasks for structured light scanning. For capturing, sldisp is run on the computer connected to the output device (e.g. monitor, projector) in order to show structured light patterns. A corresponding capture program (e.g. slcapture.py) is run on the computer connected to the input device (e.g. firewire/usb camera). By default, both computers are the same and sldisp/guppy_cap communicate over localhost via OSC UDP packets.
+mjpg-streamer is run on a computer connected to a supported camera.
+sldisp is run on a computer connected to a flatscreen or projector.
+slcapture.py talks to the two programs above to display sequences of patterns and save images of each in a directory
+slcrunch processes a directory of images into a correspondence file, or into a 3D point cloud with appropriate calibration input file
+slcalibrate processes correspondence files and outputs calibration data for cameras and projectors
 
 Typical lens calibration consists of a camera-monitor (flatscreen display) setup, with sldisp showing structured light patterns on the display. guppy_cap is used to capture image sequences, which are processed into camera-display correspondence data with slcrunch. Once you've collected enough different viewpoints of a flatscreen monitor, the correspondence data is passed to slcalibrate. Calibration output from slcalibrate includes camera matrix, camera distortion coefficients, and camera pixel dimensions.
 
@@ -65,6 +117,6 @@ Calibration output can be visualized using plotlens to see a color mapped distor
 slcalibrate can take previous calibration files for camera or projector, to reuse intrinsic calibration data. Used for camera-projector registration.
 
 
-## RPI example
+## RPI example hardware