This repository concerns the Python interface of the Flink Streaming Processing framework, called pyflink. A basic high level overview of Pyflink, largely for personal reference, is included in the repository.
While examples and documentation for using pyflink with docker may be found here, here and here (only for Flink 1.13 at the time of writing) some may benefit from seeing more examples, including examples of additional use cases.
This repository is intended to provide such further examples, the repository makes no claim to represent best Pyflink practices.
This version uses Flink 1.14 be aware that some functionality may differ between versions.
An example showing a multi-node deployment using docker swarm is included in Example 1.
(1) A simple prototype streaming pipeline for quality assessment of radio telescope data. Large example using both API's and a variety of Flink streaming operations. Multi-node deployment instructions.
[Table API, Datastream API, Stateful Function, Kafka, Elasticsearch, Kibana]
(2) The standard wordcount example for streaming and batch using both the datastream and table APIs in a docker container.
[Table API, Datastream API]
(3) A version of the playground from here with modifications to support Flink 14.4, as well as minor cosmetic changes.
[Table API, UDF, Kafka, Elasticsearch, Kibana]
(4) Example of combining the table and datastream API's to implement the keyed state function from here is included.
[Table API, Datastream API, Stateful Function, Kafka, Elasticsearch, Kibana]
(5) Another keyed state function, analysing random 1's and 0's generated and sent with kafka is included. The flink job records the longest run of consecutive 1's or 0's and the time it occurs. A kibana dashboard displaying the results is included. [Table API, Datastream API, Stateful Function, Kafka, Elasticsearch, Kibana]
It may be wise to remove and rebuild the images for (3) and (4) when switching between them.