Performing Vision

Sensing and operating is a huge part of operating a robot. One such sensing technique is Vision. There are many libraries to perform Vision calculation, and FlashLib has no need to provide another one. However, managing continuous vision operation, and outputting it to robot actions is not so simple. Thus FlashLib a simple framework to manage vision operations.

Vision Process

• Images are provided by ImageSource implementations. The source may provide the same image, different images, or not provide any image at all.
• Images then pass through Pipelines. These pipelines may perform processing or just write the image to some output.
  • For processing the image, a pipeline may be a VisionPipeline. Such pipeline processes images using Processors, and analyzes the result using an Analyzer. After an analysis was created, the analysis is passed to an output consumer.

Implementing

Basic Pipeline

The most basic process can be displaying an image from a source:

ScheduledExecutorService executorService = ...

Source<Image> source = ...
Pipeline<Image> pipeline = ...

Future<?> pollFuture = source.asyncPollAtFixedRate(executorService, Time.milliseconds(100), pipeline);

This will start an asynchronous polling of images from the source, passing it over to pipeline.

It is also possible to diverge pipes, so the images reaches multiple pipelines:

Pipeline<Image> pipeline = new SomePipeline<>()
          .divergeTo(new AnotherPipeline<>());

Processing Pipeline

Processing images is done using a Processor line. The job of the processor is to run some operations on the image and extract features for analysis. Once the processors are done, the analyzer can analyze the image for information.

To build the processing pipeline, we would use VisionPipeline, and define our processors and analyzer there:

Pipeline<Image> pipeline = new VisionPipeline.Builder<>()
        .processor((input)-> {
            // do something and return an output
            return output;
         })
        .analyse((original, postProcess)-> {
            // here we need to produce an Analysis object with information that we want
            return Optional.of(new Analysis.Builder()
                         .put("somekey", "somedata")
                         .build());    
         })
         .analysisTo((analysis)-> {
             // here we need to define what to do with the analysis, where to send it.
             // normally we can send it to the robot somehow.
         })
         .build();

Now we can use the created Pipeline like any other.

For complex processing, we can diving the processors to work in a cascading manner: each performing some operation and passing it on to the next. This can be done using the pipeTo method from Processor:

Processor<byte[], String> bytesToString = ...;
Processor<String, Integer> stringToInt = ...;

Processor<byte[], Integer> bytesToInt = bytesToString.pipeTo(stringToInt);

It is also possible to diverge the input or output of a processor into a pipeline. Can be useful to show information will processing.

Processor<byte[], String> bytesToString = ...;
Processor<String, Integer> stringToInt = ...;

Processor<byte[], Integer> bytesToInt = bytesToString
                .divergeIn(System.out::println) // will print the bytes[] received into bytesToString
                .divergeOut(System.out::println) // will print the String produced by bytesToString
                .pipeTo(stringToInt);

Using a combination of divergeIn, divergeOut and pipeTo, it is possible to create powerful, but modular processing pipelines.