Flink templates and common streaming analytics resources developed by Viseca Card Services SA

Welcome to ch.viseca.flink:flink-commons

In early 2020 we, Viseca Card Services SA, decided to share part of our efforts in the area of stream processing and Flink with the general public.

The libraries and modules presented here help us jump start Flink stream processing projects, especially with:

• bridging the initial learning curve necessary
• having a common setup for building projects that can easily be adjusted in a single place (see 'starters' further down)
• taking care of time-to-market aspects
• integration with CI/CD procedures
• and others

Feedback from 'the public' is welcome.

What it is not yet

• Well documented (although most parts have code documentation)
• Stable API: Interfaces are subject to frequent change, we don't make any guarantees for backwards compatibility
• There is no 'Getting Started' section yet
• Publication of pre-built artifacts

Subfolders description

• archetypes - Maven archtypes to create Flink Scala Jobs (all archetypes integrate the below mentioned flink-scala-commons-starter)
  • scala-only-archetype - prepares a simple Scala project
  • flink-library-archetype - prepares a Flink library used to extend functionality (not for jobs)
  • flink-job-archetype - prepares a Flink job library (without Spring Boot)
  • flink-spring-job-archetype - prepares a Flink job library including integration with Spring Boot dependency injection and configuration
  • flink-project-container-archetype - prepares a multi-module container project
• libraries - libraries to implement common stream processing concerns
  • flink-scala-commons - common extensions to the Flink ecosystem
    • ch.viseca.flink.conversion - conversion helpers
    • ch.viseca.flink.featureChain - implementation of
      • discriminated union types and
      • generalization of components for feature chains : ... the pattern was introduced at the FlinkForward SF 2017 conference
    • ch.viseca.flink.jobSetup - runtime pluggable/replaceable sources, sinks, components for Flink jobs
    • ch.viseca.flink.metaModel - extends jobSetup with a meta model
    • ch.viseca.flink.logging - logging integration
    • ch.viseca.flink.operators - common operator functions
    • ch.viseca.flink.waterMarkAssignment - watermark assigners for streamed batch sources
• samples - sample modules to demonstrate principles and for documentation
  • flink-samples-featureChain-noDep - usage example for a feature chain and discriminated unions
  • flink-samples-spring-featureChain - the same example with Spring Boot integration
• starters - optional parent POMs for a common build/dependency setup
  • flink-scala-commons-starter - a parent POM module
    • that configures build processes and dependencies by means of a set of Maven profiles
    • thus Maven profiles are enabled using tag-files (look for the 'inheritedProfiles' folders)
    • by deriving a project from the starter POM you subscribe to a tested set of build instructions and common dependency versions,
    • for changing to newer versions of e.g. Flink it suffices to derive from a different versions of the start
    • the same is true for corrections in the build processes - there is no need to adjust all involved projects