Skip to content

Efrat19/flink-ads-processing

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

flink-ads-processing

A flink + kotlin exercise

Explore

Start the project

docker compose up

this will take a while, go explore flink windowing docs meanwhile.

Once the stack is up, you can

What's in here

flowchart TB
        ws[Web Crawler] --> tsa>topic:\n scraped-ads]
        tsa --- fj1[Flink job:\n AdsByCityYearTumbling]
        tsa --- fj2[Flink job:\n AdsByCitySliding]
        fj1 --> abctw>topic:\n ads-by-city-and-year-tmbl-win]
        fj2 --> abcsw>topic:\n ads-by-city-sld-win]
        abctw --- g[visualization stack]
        abcsw --- g
Loading

Stack Components:

  1. A scrapy crawler for collecting ads from a board website
  2. A kraft deployment - stores the raw scraped-ads topic, and the output topics from flink aggregation jobs.
  3. A Flink cluster + client, to submit and run 2 kotlin aggregation jobs.
  4. The Visualization stack - promtail, loki and grafana (honestly I wanted grafana to read directly from kafka but the required plugin relies on librdkafka and won't compile on arm64 😢)

Helpful links (i.e. My code is copied from)

About

A flink + kotlin exercise

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published