Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0b991d0
feat: pluggable external LLM providers (model_type: external)
psriramsnc Nov 17, 2026
e92dd6e
feat: raise ModelServerDownError instead of sys.exit() in library runs
psriramsnc Nov 17, 2026
59c669f
feat: async Workflow.arun() for running inside an existing event loop
psriramsnc Nov 17, 2026
25882f9
feat: isolate per-run and per-request state for concurrent workflows
psriramsnc Nov 17, 2026
17e5f98
feat: don't load .env or override os.environ in library runs
psriramsnc Nov 17, 2026
4c526d9
fix: keep CLI signal handling unchanged; no process-wide handlers in …
psriramsnc Nov 17, 2026
b98557d
feat: return RunResult from Workflow.run()/arun() with usage, failure…
psriramsnc Nov 17, 2026
412e7aa
fix: write results not yet checkpointed at the end of non-resumable runs
psriramsnc Nov 17, 2026
ce2f623
feat: build workflows from config dicts with Workflow.from_config
psriramsnc Nov 17, 2026
baf28d5
feat: accept hooks as Python objects and publish sygra.hooks as publi…
psriramsnc Nov 17, 2026
857078e
feat: in-memory sources, stable record ids and runs without output files
psriramsnc Nov 17, 2026
e18501e
feat: model roles resolved per run (Workflow.model_roles, run(model_r…
psriramsnc Nov 17, 2026
eebfb2a
docs: self-contained library example run from plain Python with FakeP…
psriramsnc Nov 17, 2026
bec8449
build: support Python 3.12/3.13 and widen web-stack dependency ranges
psriramsnc Nov 17, 2026
a3b8fab
ci: fix the Python 3.9 runner and the fasttext-wheel build on Python …
psriramsnc Nov 18, 2026
3811b05
chore: address code-quality review comments
psriramsnc Nov 18, 2026
a0b04ab
Restore the required 'build' CI check after the Python matrix change
psriramsnc Nov 18, 2026
4bee5e6
Build releases with Python 3.13 in the PyPI publish workflow
psriramsnc Nov 18, 2026
5a52c2a
build: allow pysnc 1.2
psriramsnc Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,22 @@ permissions:

jobs:

build:
test:
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# Lowest and highest supported versions; pinned patches keep the cached venv valid.
# Python 3.9 builds exist only up to Ubuntu 24.04.
include:
- python-version: "3.9.25"
os: ubuntu-24.04
- python-version: "3.13.13"
os: ubuntu-latest
env:
UV_PROJECT_ENVIRONMENT: .venv
UV_PYTHON: ${{ matrix.python-version }}

steps:
# ---- Disk Cleanup (Safe) ----
Expand All @@ -40,7 +51,7 @@ jobs:
- uses: actions/setup-python@v6
id: setup-python
with:
python-version: "3.9.25"
python-version: ${{ matrix.python-version }}

# ---- Install uv ----
- name: Install uv
Expand Down Expand Up @@ -87,15 +98,26 @@ jobs:
- name: Run tests
run: make test

# ---- DOCS ----
# ---- DOCS (once, on the lowest version) ----
- name: Build docs
if: matrix.python-version == '3.9.25'
run: make docs

# ---- Upload docs artifact ----
- uses: actions/upload-pages-artifact@v4
if: matrix.python-version == '3.9.25'
with:
path: site

# ---- Required status check: passes only if every matrix job passed ----
build:
if: always() && (github.event_name != 'pull_request' || github.event.pull_request.draft == false)
needs: test
runs-on: ubuntu-latest
steps:
- name: Check matrix result
run: test "${{ needs.test.result }}" = "success"

# ---- Deploy Docs ----
deploy_docs:
if: github.ref == 'refs/heads/main'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ on:
jobs:
publish:
runs-on: ubuntu-latest
env:
UV_PYTHON: "3.13"

steps:
- name: Checkout repository
Expand All @@ -16,7 +18,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.9"
python-version: "3.13"

- name: Install uv
run: |
Expand Down
125 changes: 125 additions & 0 deletions docs/features/external_llm_providers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# External LLM Providers

> **Route a model's calls through your own client with `model_type: external`**

## Overview

By default SyGra calls models through its built-in HTTP and LiteLLM clients, configured with `SYGRA_<MODEL>_URL` / `SYGRA_<MODEL>_TOKEN`. An **external provider** lets the application that runs SyGra serve the calls instead.

This is useful when:

- Calls must go through your own gateway, router or in-house SDK, with your own auth.
- You want to add caching, recording, rate limiting or cost tracking around every call.
- You need a deterministic test double, so pipelines can be tested without network access.

A provider is any object with an async `generate` method:

```python
from sygra import LLMRequest, ModelResponse


class MyProvider:
async def generate(self, request: LLMRequest) -> ModelResponse:
text = await my_client.complete(request.messages, model=request.model, **request.parameters)
return ModelResponse(llm_response=text, response_code=200)
```

**The provider owns transport, authentication, retries and concurrency.** SyGra builds the request, calls the provider, and then handles the response as usual: structured-output fallback, post-processors and failure detection.

## Quick Start

```python
from sygra import FakeProvider, Workflow

provider = FakeProvider(lambda req: req.messages[-1]["content"].upper())

records = (
Workflow("external_demo")
.source([{"id": "1", "text": "hello"}, {"id": "2", "text": "world"}])
.llm({"name": "generator", "model_type": "external"}, "{text}", output="answer")
.llm_provider(provider) # serves model_type: external for this run only
.run(num_records=2)
)
# records[i]["answer"] == "HELLO" / "WORLD"
```

## Model configuration

Any model config (in a task's `graph_config.yaml`, in `models.yaml`, or passed in code) can use an external provider:

```yaml
model:
name: generator # SyGra model name
model_type: external
provider: my_gateway # registered provider name (default: "default")
model: acme-large # identifier passed to the provider (default: name)
parameters: # passed through as LLMRequest.parameters
temperature: 0.7
max_tokens: 2048
provider_options: # free-form, passed through as LLMRequest.provider_options
route: eu
```

| Key | Default | Meaning |
|---|---|---|
| `provider` | `default` | Name of the registered provider that serves this model. |
| `model` | `name` | Model identifier sent to the provider. |
| `parameters` | `{}` | Generation parameters. If a provider can't apply one, it should ignore it. |
| `provider_options` | `{}` | Provider-specific options. SyGra doesn't interpret them. |
| `retry_attempts` | `1` | SyGra-side attempts. The default `1` means no SyGra retries, because the provider owns retries. |
| `delay` | `0` | Pre-call delay in ms. |

## Registering providers

| API | Scope |
|---|---|
| `Workflow.llm_provider(provider, name="default")` | Only for that workflow's `run()` / `arun()`, in the calling context. |
| `with use_llm_providers({"name": provider}): ...` | Code (and asyncio tasks) started inside the block. Shadows global registrations. |
| `register_llm_provider(provider, name="default")` | Process-wide. |

**Resolution order:** context-scoped registrations first, then process-wide ones. A model whose provider can't be found raises `LookupError` when the graph is built, before any record is processed.

**Concurrency:** scoped registrations are isolated per thread and per context, so two workflows running at the same time can use different providers.

## Request and response

`LLMRequest` fields:

| Field | Type | Description |
|---|---|---|
| `name` | `str` | SyGra model name. |
| `model` | `str` | Model identifier to request. |
| `messages` | `list[dict]` | OpenAI-style chat messages (`role`, `content`, plus `tool_calls` / `tool_call_id` where present). |
| `parameters` | `dict` | Generation parameters. |
| `tools` | `list[dict]` | Node tools in the OpenAI function-tool format; empty if the node has none. |
| `tool_choice` | `Any` | The node's tool choice, when tools are used. |
| `provider_options` | `dict` | Pass-through options from the model config. |

Return a `ModelResponse`:

- `llm_response`: the text reply.
- `response_code`: `200` on success. Any other code marks the call as failed: the node records a failure and the record is handled like any other model error.
- `tool_calls`: tool calls in OpenAI dict format, if any.
- `usage`: optional `TokenUsage(prompt_tokens, completion_tokens, total_tokens, estimated)`. It feeds SyGra's metadata and cost tracking. Set `estimated=True` when your backend doesn't report counts and you computed them yourself.
- `served_model`: optional. The model that actually answered. Set it when your backend can route or fall back to a different model than the one requested.

**Exceptions:** an exception raised by `generate` is caught. It becomes an error response that uses the exception's `status_code` attribute when present, and `999` otherwise.

**Server-down detection:** codes 404, 500, 501, 502 and 503 count toward SyGra's [server-down check](../getting_started/model_configuration.md#server-down-handling), like they do for built-in models. Set `server_down_action: ignore` if your provider handles outages itself.

## Testing with `FakeProvider`

`FakeProvider` returns a fixed string, or the result of a callable, and records every request it receives:

```python
from sygra import FakeProvider, TokenUsage

fake = FakeProvider("ok", usage=TokenUsage(prompt_tokens=10, completion_tokens=2, total_tokens=12))
...
assert fake.requests[0].messages[-1]["content"] == "expected prompt"
```

## Limitations

- Agent nodes (`node_type: agent`) use the LangGraph chat-model backend and do not support `model_type: external` yet.
- Native structured output isn't delegated to the provider. Structured output for external models uses SyGra's instruction-based fallback: it adds format instructions to the prompt and validates the reply.
2 changes: 2 additions & 0 deletions docs/features/metadata_tracking.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ print(f"Models used: {list(metadata['models'].keys())}")
print(f"Nodes executed: {list(metadata['nodes'].keys())}")
```

**With `Workflow.run()` / `arun()`:** each run collects into its own collector, so concurrent runs in one process don't mix their metrics. While a run is active, `get_metadata_collector()` returns that run's collector. After the run, the process-wide collector holds the metadata of the run that finished last.

---

## Configuration
Expand Down
34 changes: 33 additions & 1 deletion docs/getting_started/model_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,16 @@ For `mixtral_8x7b` and `gpt4`, set:

You should use a `.env` file at the project root or set these in your shell environment.

**Where `.env` is loaded:**
- **CLI:** `main.py` loads `.env` from the working directory at startup, as before.
- **Library** (`Workflow.run()` / `arun()`): SyGra doesn't load `.env` on its own, and it never overrides variables already set in `os.environ`. To load one, pass it to the run:

```python
workflow.run(num_records=10, env_file=".env") # existing variables win
```

Alternatively, put connection settings straight into the model config instead of using `SYGRA_*` variables: `{"name": ..., "model_type": ..., "url": ..., "auth_token": ...}`. Values in the model config take precedence over environment variables.

**Note:**
If you want to define a list of URLs for any model, you can use pipe (`|`) as a separator. For example, if you have a model called `mixtral_8x7b` with URLs `https://myserver/models/mixtral-8x7b` and `https://myserver/models/mixtral-8x7b-2`, you can set the following environment variables as shown in examples below.

Expand Down Expand Up @@ -78,10 +88,33 @@ SYGRA_MIXTRAL_8X7B_CHAT_TEMPLATE={% for m in messages %} ... {% endfor %}
| `ssl_cert` | *(Optional)* Path to SSL certificate file |
| `json_payload` | *(Optional)* Boolean: use JSON payload instead of JSON string for `http client` based models (default: false) |
| `headers` | *(Optional)* Dictionary of headers to be sent with the request for `http client` based models |
| `server_down_action` | *(Optional)* What to do when the model looks down (see [Server-down handling](#server-down-handling)): `exit`, `raise` or `ignore`. Default: `exit` for CLI runs, `raise` for `Workflow.run()` |
![Note](https://img.shields.io/badge/Note-important-yellow)
> - Do **not** include `url`, `auth_token`, or `api_key` in your YAML config. These are sourced from environment variables as described above.<br>
> - If you want to set **ssl_verify** to **false** globally, you can set `ssl_verify:false` under `model_config` section in config/configuration.yaml

#### Server-down handling

SyGra treats a model as down when it returns 10 server-down responses (404, 500, 501, 502 or 503) within 30 seconds. What happens next depends on the action:

| Action | Behavior |
|---|---|
| `exit` | Log an error and exit the process. This is the default for CLI runs (`main.py`). |
| `raise` | Stop the run and raise `sygra.ModelServerDownError`. In-flight records are cancelled, and the process keeps running. This is the default inside `Workflow.run()`. |
| `ignore` | Don't track server-down responses for this model. Failed calls are still recorded as failed records. |

The model's `server_down_action` key wins. Otherwise the action set for the current context applies, then `exit`. To get `raise` behavior outside `Workflow.run()`, for example when you drive a task executor yourself, use the context manager:

```python
from sygra import ModelServerDownError, server_down_action

with server_down_action("raise"):
try:
executor.execute()
except ModelServerDownError as err:
print(err.model_name, err.failures, err.window_seconds)
```

#### Customizable Model Parameters

- `temperature`: Sampling randomness (0.0–2.0; lower is more deterministic)
Expand Down Expand Up @@ -234,4 +267,3 @@ gemini_2.5_proxy:
temperature: 0
```
---

Loading
Loading