Skip to content

Commit d4ee61b

Browse files
sionsmithclaude
sionsmith
and
claude
committed
Initial release: K2I Kafka to Iceberg streaming ingestion
Features: - Kafka consumer with smart backpressure and retry logic - Hot buffer with Apache Arrow for sub-second query freshness - Iceberg writer supporting REST, Hive, Glue, and Nessie catalogs - Transaction log for crash recovery and exactly-once semantics - Automatic maintenance: compaction, snapshot expiration, orphan cleanup - Prometheus metrics and health check endpoints - CLI commands: ingest, status, maintenance, validate Documentation: - Comprehensive user guides (quickstart, configuration, deployment) - Architecture documentation with design details - CLI command reference - Troubleshooting guide CI/CD: - GitHub Actions for testing and release - cargo-dist integration for cross-platform builds - Docker image publishing - Homebrew formula publishing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
0 parents  commit d4ee61b

77 files changed

Lines changed: 39777 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/bug.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: Bug report
2+
description: Report a bug or unexpected behavior
3+
4+
labels: ["bug"]
5+
6+
body:
7+
- type: markdown
8+
attributes:
9+
value: |
10+
Thanks for reporting an issue! Please provide as much detail as possible to help us reproduce and fix the problem.
11+
- type: textarea
12+
attributes:
13+
label: Describe the bug
14+
description: A clear and concise description of what the bug is.
15+
placeholder: Describe the issue step by step
16+
validations:
17+
required: true
18+
- type: textarea
19+
attributes:
20+
label: Steps to reproduce
21+
description: Steps to reproduce the behavior.
22+
placeholder: |
23+
1. Run command '...'
24+
2. With config '...'
25+
3. See error
26+
validations:
27+
required: true
28+
- type: textarea
29+
attributes:
30+
label: Expected behavior
31+
description: A clear description of what you expected to happen.
32+
validations:
33+
required: false
34+
- type: textarea
35+
attributes:
36+
label: Error output / Logs
37+
description: If applicable, paste the error message or relevant logs.
38+
render: shell
39+
validations:
40+
required: false
41+
- type: textarea
42+
attributes:
43+
label: Configuration
44+
description: Share your (sanitized) configuration file if relevant.
45+
render: toml
46+
validations:
47+
required: false
48+
- type: textarea
49+
attributes:
50+
label: Environment
51+
description: Environment information where the problem occurs.
52+
value: |
53+
- k2i version:
54+
- OS:
55+
- Kafka version:
56+
- Iceberg catalog type: [REST / Hive / Glue / Nessie]
57+
- Storage backend: [S3 / GCS / Azure Blob / filesystem]
58+
validations:
59+
required: false
60+
- type: textarea
61+
attributes:
62+
label: Additional context
63+
description: Add any other context about the problem here.
64+
validations:
65+
required: false

‎.github/ISSUE_TEMPLATE/config.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
blank_issues_enabled: false
2+
contact_links:
3+
- name: Documentation
4+
url: https://github.com/osodevops/k2i#readme
5+
about: Check the documentation before opening an issue

‎.github/ISSUE_TEMPLATE/feature.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Feature request
2+
description: Suggest a new feature or improvement
3+
4+
labels: ["enhancement"]
5+
6+
body:
7+
- type: markdown
8+
attributes:
9+
value: |
10+
Thanks for your feature suggestion! Please describe your idea in detail.
11+
- type: textarea
12+
attributes:
13+
label: Is your feature request related to a problem?
14+
description: A clear description of what the problem is.
15+
placeholder: I'm always frustrated when [...]
16+
validations:
17+
required: false
18+
- type: textarea
19+
attributes:
20+
label: Describe the solution you'd like
21+
description: A clear description of what you want to happen.
22+
validations:
23+
required: true
24+
- type: textarea
25+
attributes:
26+
label: Describe alternatives you've considered
27+
description: Any alternative solutions or features you've considered.
28+
validations:
29+
required: false
30+
- type: textarea
31+
attributes:
32+
label: Use case
33+
description: Describe the use case or scenario where this feature would be helpful.
34+
validations:
35+
required: false
36+
- type: textarea
37+
attributes:
38+
label: Additional context
39+
description: Add any other context or screenshots about the feature request.
40+
validations:
41+
required: false

‎.github/workflows/docker-publish.yml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
name: Build & Push Docker Image
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*'
7+
8+
env:
9+
REGISTRY: ghcr.io
10+
IMAGE_NAME: osodevops/k2i
11+
CARGO_TERM_COLOR: always
12+
13+
jobs:
14+
# Run tests first - must pass before Docker build
15+
test:
16+
name: Run Tests
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v4
20+
21+
- name: Install system dependencies
22+
run: |
23+
sudo apt-get update
24+
sudo apt-get install -y cmake libssl-dev libsasl2-dev pkg-config
25+
26+
- name: Install Rust toolchain
27+
uses: dtolnay/rust-toolchain@stable
28+
with:
29+
components: clippy, rustfmt
30+
31+
- name: Cache cargo registry
32+
uses: actions/cache@v4
33+
with:
34+
path: |
35+
~/.cargo/registry
36+
~/.cargo/git
37+
target
38+
key: ${{ runner.os }}-cargo-docker-${{ hashFiles('**/Cargo.lock') }}
39+
restore-keys: |
40+
${{ runner.os }}-cargo-docker-
41+
42+
- name: Check formatting
43+
run: cargo fmt --all -- --check
44+
45+
- name: Run clippy
46+
run: cargo clippy --all-targets --all-features -- -D warnings
47+
48+
- name: Run unit tests
49+
run: cargo test --lib --all-features
50+
env:
51+
RUST_LOG: debug
52+
53+
build-and-push:
54+
needs: test
55+
runs-on: ubuntu-latest
56+
permissions:
57+
contents: read
58+
packages: write
59+
60+
steps:
61+
- name: Checkout repository
62+
uses: actions/checkout@v4
63+
64+
- name: Set up Docker Buildx
65+
uses: docker/setup-buildx-action@v3
66+
67+
- name: Log in to GitHub Container Registry
68+
uses: docker/login-action@v3
69+
with:
70+
registry: ${{ env.REGISTRY }}
71+
username: ${{ github.actor }}
72+
password: ${{ secrets.GITHUB_TOKEN }}
73+
74+
- name: Extract metadata
75+
id: meta
76+
uses: docker/metadata-action@v5
77+
with:
78+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
79+
tags: |
80+
type=semver,pattern=v{{version}}
81+
type=semver,pattern=v{{major}}.{{minor}}
82+
type=raw,value=latest
83+
84+
- name: Build and push Docker image
85+
uses: docker/build-push-action@v5
86+
with:
87+
context: .
88+
push: true
89+
tags: ${{ steps.meta.outputs.tags }}
90+
labels: ${{ steps.meta.outputs.labels }}
91+
cache-from: type=gha
92+
cache-to: type=gha,mode=max

0 commit comments

Comments
 (0)