Apache Pinot Recipes

This repository contains Apache Pinot recipes, many referenced in StarTree developer site.

(🍷) identifies recipes that have been recently audited and updated.

Debezium CDC

• Postgres🍷

General Recipes

• DateTime Strings to Timestamps
• Understanding Lookup-based Join Support🍷
• Infer schema from JSON data
• Time boundary in hybrid tables🍷

Minio / Deepstore

• Using MinIO as Deep Store for an Offline Table🍷
• Minio Real-Time🍷

Batch Data Ingestion

• Importing JSON files
• Importing CSV files with columns containing spaces
• Import Data files from different directories
• Ingest Parquet Files from a S3 Bucket into Pinot Using Spark

Streaming Ingestion

• Ingest Avro from Kafka
• Ingest JSON files from Kafka🍷
• Pulsar🍷

Upserts

• Full Upserts🍷
• Partial Upserts

Transformation Functions

• Groovy Transformation Functions
• JSON Transformation Functions
• Chaining Transformation Functions

Real-Time to Offline Job (RT2OFF)

• Manually schedule RT2OFF job🍷
• Automatically schedule RT2OFF job🍷
• Backfill segment created by RT2OFF job

Geospatial

• Storing Geospatial objects
• Geospatial indexes
• Default values for Geospatial fields

Indexes

• JSON indexes
• Update JSON index
• StarTree Index🍷

Merge and Rollup

• Merge small segments🍷
• Automatically merge real-time segments

Querying

• Query by segment name

Datasets

• Chicago Crimes

Operational

• Force Commit
• Pause/Resume consumption
• Checking segment to server assignment
• Removing server

Artificial Intelligence / Similarity Search

• Celebrity Lookalike🍷
• GenAI🍷
• Vector🍷
• Computer Vision🍷