GitHub - plabayo/rama: modular service framework to move and transform network packets

🦙 Rama (ラマ) is a modular service framework for the 🦀 Rust language to move and transform your network packets. The reasons behind the creation of rama can be read in the "Why Rama" chapter.

Rama is async-first using Tokio as its only Async Runtime. Please refer to the examples found in the /examples dir to get inspired on how you can use it for your purposes.

💡 If your organization relies on Rama (ラマ) for its operations, we invite you to consider becoming a sponsor 💖. By supporting our project, you'll help ensure its continued development and success. To learn more about sponsorship opportunities, please refer to the "Sponsors" section below or contact us directly at [email protected].

This framework comes with 🔋 batteries included, giving you the full freedome to build the middleware and services you want, without having to repeat the "common":

category	support list
✅ transports	✅ tcp ⸱ 🏗️ udp ⁽²⁾ ⸱ ✅ middleware
✅ http	✅ auto ⸱ ✅ http/1.1 ⸱ ✅ h2 ⸱ 🏗️ h3 ⁽²⁾ ⸱ ✅ middleware
✅ web server	✅ fs ⸱ ✅ redirect ⸱ ✅ dyn router ⸱ ✅ static router ⸱ ✅ handler extractors ⸱ ✅ k8s healthcheck
✅ http client	✅ client ⸱ ✅ high level API ⸱ ✅ Proxy Connect ⸱ ❌ Chromium Http ⁽³⁾
✅ tls	✅ Rustls ⸱ ✅ BoringSSL ⸱ ❌ NSS ⁽³⁾
✅ dns	✅ DNS Resolver
✅ proxy protocols	✅ PROXY protocol ⸱ ✅ http proxy ⸱ ✅ https proxy ⸱ 🏗️ SOCKS5 ⁽¹⁾ ⸱ 🏗️ SOCKS5H ⁽¹⁾
🏗️ web protocols	🏗️ Web Sockets (WS) ⁽²⁾ ⸱ 🏗️ WSS ⁽²⁾ ⸱ ❌ Web Transport ⁽³⁾ ⸱ ❌ gRPC ⁽³⁾
✅ async-method trait services	✅ Service ⸱ ✅ Layer ⸱ ✅ context ⸱ ✅ dyn dispatch ⸱ ✅ middleware
✅ telemetry	✅ tracing ⸱ ✅ opentelemetry ⸱ ✅ http metrics ⸱ ✅ transport metrics
✅ upstream proxies	✅ MemoryProxyDB ⸱ ✅ L4 Username Config ⸱ ✅ Proxy Filters
🏗️ User Agent (UA)	🏗️ Http Emulation ⁽¹⁾ ⸱ 🏗️ Tls Emulation ⁽¹⁾ ⸱ ✅ UA Parsing
✅ utilities	✅ error handling ⸱ ✅ graceful shutdown ⸱ 🏗️ Connection Pool ⁽²⁾ ⸱ 🏗️ IP2Loc ⁽²⁾
🏗️ TUI	🏗️ traffic logger ⁽²⁾ ⸱ 🏗️ curl export ⁽²⁾ ⸱ ❌ traffic intercept ⁽³⁾ ⸱ ❌ traffic replay ⁽³⁾
✅ binary	✅ prebuilt binaries ⸱ 🏗️ proxy config ⁽²⁾ ⸱ ✅ http client ⸱ ❌ WASM Plugins ⁽³⁾
🏗️ data scraping	🏗️ Html Processor ⁽²⁾ ⸱ ❌ Json Processor ⁽³⁾
❌ browser	❌ JS Engine ⁽³⁾ ⸱ ❌ Web API Emulation ⁽³⁾

🗒️ Footnotes

⁽¹⁾ Part of v0.2.0 milestone (ETA: 2024 mid Q3)

⁽²⁾ Part of v0.3.0 milestone (ETA: 2024 end Q3)

⁽³⁾ No immediate plans, but on our radar. Please open an issue to request this feature if you have an immediate need for it. Please add sufficient motivation/reasoning and consider becoming a sponsor to help accelerate its priority.

The primary focus of Rama is to aid you in your development of proxies:

🚦 Reverse proxies;
🔓 TLS Termination proxies;
🌐 HTTP(S) proxies;
🧦 SOCKS5 proxies (will be implemented in v0.3);
🔎 MITM proxies;
🕵️‍♀️ Distortion proxies.

💡 Check out the "Intro to Proxies" chapters in the Rama book to learn more about the different kind of proxies. It might help in case you are new to developing proxies.

The Distortion proxies support comes with User Agent (UA) emulation capabilities. The emulations are made possible by patterns and data extracted using rama-fp. The service is publicly exposed at https://fp.ramaproxy.org, made possible by our sponsor host https://fly.io/.

🔁 https://echo.ramaproxy.org/ is another service publicly exposed. In contrast to the Fingerprinting Service it is aimed at developers and allows you to send any http request you wish in order to get an insight on the Tls Info and Http Request Info the server receives from you when making that request.
curl -XPOST 'https://echo.ramaproxy.org/foo?bar=baz' \
  -H 'x-magic: 42' --data 'whatever forever'
Feel free to make use of while crafting distorted http requests, but please do so with moderation. In case you have ideas on how to improve the service, please let us know by opening an issue.

Using the rama binary you can also run both the echo and fp service yourself, locally or as an external facing web service.

Please run your own echo service instead of using echo.ramaproxy.org in case you are planning to send a lot of traffic to the echo service.

BrowserStack sponsors Rama by providing automated cross-platform browser testing on real devices, which uses the public fingerprinting service to aid in automated fingerprint collection on both the Http and Tls layers. By design we do not consider Tcp and Udp fingerprinting.

Next to proxies, Rama can also be used to develop Web Services and Http Clients.

Learn more by reading the Rama book at https://ramaproxy.org/book;
or checkout the framework Rust docs at https://docs.rs/rama;
- edge docs (for main branch) can be found at https://ramaproxy.org/docs/rama.

📖 Rama's full documentation, references and background material can be found in the form of the "rama book" at https://ramaproxy.org/book.

💬 Come join us at Discord on the #rama public channel. To ask questions, discuss ideas and ask how rama may be useful for you.

⌨️ | `rama` binary

The rama binary allows you to use a lot of what rama has to offer without having to code yourself. It comes with a working http client for CLI, which emulates User-Agents and has other utilities. And it also comes with IP/Echo services.

It also allows you to run a rama proxy, configured to your needs.

Learn more about the rama binary and how to install it at https://ramaproxy.org/book/binary/rama.

🧪 | Experimental

🦙 Rama (ラマ) is to be considered experimental software for the foreseeable future. In the meanwhile it is already used in production by ourselves and others alike. This is great as it gives us new perspectives and data to further improve and grow the framework. It does mean however that there are still several non-backward compatible releases that will follow 0.2.

In the meanwhile the async ecosystem of Rust is also maturing, and edition 2024 is also to be expected as a 2024 end of year gift. It goes also without saying that we do not nilly-willy change designs or break on purpose. The core design is by now also well defined. But truth has to be said, there is still plenty to be improve and work out. Production use and feedback from you and other users helps a lot with that. As such, if you use Rama do let us know feedback over Discord, email or a GitHub issue.

👉 If you are a company or enterprise that makes use of Rama, or even an individual user that makes use of Rama for commcercial purposes. Please consider becoming a business/enterprise subscriber. It helps make the development cycle to remain sustainable, and is beneficial to you as well. As part of your benefits we are also available to assist you with migrations between breaking releases. For enterprise users we can even make time to develop those PR's in your integration codebases ourselves on your behalf. A win for everybody. 💪

📣 | Rama Ecosystem

For now there are only the rama crates found in this repository, also referred to as "official" rama crates.

We welcome however community contributions not only in the form of contributions to this repository, but also have people write their own crates as extensions to the rama ecosystem. E.g. perhaps you wish to support an alternative http/tls backend.

In case you have ideas for new features or stacks please let us know first. Perhaps there is room for these within an official rama crate. In case it is considered out of scope you are free to make your own community rama crate. Please prefix all rama community crates with "rama-x", this way the crates are easy to find, and are sufficiently different from "official" rama crates".

Once you have such a crate published do let us know it, such that we can list them here.

📦 | Rama Crates

The rama crate can be used as the one and only dependency. However, as you can also read in the "DIY" chapter of the book at https://ramaproxy.org/book/diy.html#empowering, you are able to pick and choose not only what specific parts of rama you wish to use, but also in fact what specific (sub) crates.

Here is a list of all rama crates:

rama: one crate to rule them all
rama-error: error utilities for rama and its users
rama-macros: contains the procedural macros used by rama
rama-utils: utilities crate for rama
rama-core: core crate containing the service, layer and context used by all other rama code, as well as some other core utilities
rama-net: rama network types and utilities
rama-dns: DNS support for rama
rama-tcp: TCP support for rama
rama-udp: UDP support for rama
rama-tls: TLS support for rama (types, rustls and boring)
rama-proxy: proxy types and utilities for rama
rama-socks5: SOCKS5 support for rama
rama-haproxy: rama HaProxy support
rama-ua: User-Agent (UA) support for rama
rama-http-types: http types and utilities
rama-http: rama http services, layers and utilities
rama-http-backend: default http backend for rama
rama-http-core: http protocol implementation driving rama-http-backend

🏢 | Proxy Examples

/examples/tls_termination.rs: Spawns a mini handmade http server, as well as a TLS termination proxy, forwarding the plain text stream to the first.
/examples/tls_termination.rs: Spawns a mini handmade http server, as well as a TLS termination proxy, forwarding the plain text stream to the first.
/examples/mtls_tunnel_and_service.rs: Example of how to do mTls (manual Tls, where the client also needs a certificate) using rama, as well as how one might use this concept to provide a tunnel service build with these concepts;
/examples/http_connect_proxy.rs: Spawns a minimal http proxy which accepts http/1.1 and h2 connections alike, and proxies them to the target host.

🌐 | Web Services

Developing proxies are the primary focus of Rama (ラマ). It can however also be used to develop web services to serve web pages, Http API's and static content. This comes with many of the same benefits that you get when developing proxies using Rama:

Use Async Method Traits;
Reuse modular Tower-like middleware using extensions as well as strongly typed state;
Have the ability to be in full control of your web stack from Transport Layer (Tcp, Udp), through Tls and Http;
If all you care about is the Http layer then that is fine to.
Be able to trust that your incoming Application Http data has not been modified (e.g. Http header casing and order is preserved);
Easily develop your service at a Request layer and High level functions alike, choices are yours and can be combined.

Examples of the kind of web services you might build with rama in function of your proxy service:

a k8s health service (/examples/http_k8s_health.rs);
a metric exposure service;
a minimal api service (e.g. to expose device profiles or certificates);
a graphical interface / control panel;

📖 Learn more about developing web services in the Rama book: https://ramaproxy.org/book/web_servers.html.

🌐 | Web Service Examples

Here are some low level web service examples without fancy features:

/examples/http_listener_hello.rs: is the most basic example on how to provide a root service with no needs for endpoints or anything else (e.g. good enough for some use cases related to health services or metrics exposures);
- /examples/http_health_check.rs is an even more minimal example of a health check service returning a 200 OK for any incoming request.
/examples/http_service_hello.rs: is an example similar to the previous example but shows how you can also operate on the underlying transport (TCP) layer, prior to passing it to your http service;

There's also a premade webservice that can be used as the health service for your proxy k8s workloads:

/examples/http_k8s_health.rs: built-in web service that can be used as a k8s health service for proxies deploying as a k8s deployment;

The following are examples that use the high level concepts of Request/State extractors and IntoResponse converters, that you'll recognise from axum, just as available for rama services:

/examples/http_key_value_store.rs: a web service example showcasing how one might do a key value store web service using Rama;
/examples/http_web_service_dir_and_api.rs: a web service example showcasing how one can make a web service to serve a website which includes an XHR API;

For a production-like example of a web service you can also read the rama cli fp cmd source code. This is the webservice behind the Rama fingerprinting service, which is used by the maintainers of 🦙 Rama (ラマ) to generate the UA emulation data for the Http and TLS layers. It is not meant to fingerprint humans or users. Instead it is meant to help automated processes look like a human.

💡 This example showcases how you can make use of the match_service macro to create a Box-free service router. Another example of this approach can be seen in the http_service_match.rs example.

🧑‍💻 | Http Clients

In The rama book you can read and learn that a big pillar of Rama's architecture is build on top of the Service concept. A Service takes as input a user-defined State (e.g. containing your database Pool) and a Request, and uses it to serve either a Response or Error. Such a Service can produce the response "directly" (also called ☘️ Leaf services) or instead pass the request and state to an inner Service which it wraps around (so called 🍔 Middlewares).

It's a powerful concept, originally introduced to Rust by the Tower ecosystem and allows you build complex stacks specialised to your needs in a modular and easy manner. Even cooler is that this works for both clients and servers alike.

Rama provides an HttpClient which sends your Http Request over the network and returns the Response if it receives and read one or an Error otherwise. Combined with the many Layers (middleware) that Rama provides and perhaps also some developed by you it is possible to create a powerful Http client suited to your needs.

As a 🍒 cherry on the cake you can import the HttpClientExt trait in your Rust module to be able to use your Http Client Service stack using a high level API to build and send requests with ease.

🧑‍💻 | Http Client Example

💡 The full "high level" example can be found at /examples/http_high_level_client.rs.

use rama::http::service::client::HttpClientExt;

let client = (
    TraceLayer::new_for_http(),
    DecompressionLayer::new(),
    AddAuthorizationLayer::basic("john", "123")
        .as_sensitive(true)
        .if_not_present(),
    RetryLayer::new(
        ManagedPolicy::default().with_backoff(ExponentialBackoff::default()),
    ),
).layer(HttpClient::default());

#[derive(Debug, Deserialize)]
struct Info {
    name: String,
    example: String,
    magic: u64,
}

let info: Info = client
    .get("http://example.com/info")
    .header("x-magic", "42")
    .typed_header(Accept::json())
    .send(Context::default())
    .await
    .unwrap()
    .try_into_json()
    .await
    .unwrap();

⛨ | Safety

The rama crates avoid unsafe_code, but do make use of it for some low level primitives (e.g. http core) or indirectly because of bindgens to C (e.g. boring).

We also make use of cargo vet to audit our supply chain.

🦀 | Compatibility

Rama (ラマ) is developed mostly on MacOS M-Series machines and run in production on a variety of Linux systems. There is no windows support, and neither do we test on that platform.

platform	tested	test platform
MacOS	✅	MacOS Apple Silicon (developer laptop) and macos-12 Intel (GitHub Action)
Linux	✅	Ubuntu 22.04 (GitHub Action)

Please open a ticket in case you have compatibility issues for your setup/platform. Our goal is not to support all possible platformns in the world, but we do want to support as many as we reasonably can.

Minimum supported Rust version

Rama's MSRV is 1.80.

Using GitHub Actions we also test if rama on that version still works on the stable and beta versions of rust as well.

🧭 | Roadmap

Please refer to https://github.com/plabayo/rama/milestones to know what's on the roadmap. Is there something not on the roadmap for the next version that you would really like? Please create a feature request to request it and become a sponsor if you can.

📰 | Media Appearances

Rama (0.2) was featured in a 📻 Rustacean episode on the 19th of May 2024, and available to listen at https://rustacean-station.org/episode/glen-de-cauwsemaecker/. In this episode Glen explains the history of Rama, why it exists, how it can be used and more.

💼 | License

This project is dual-licensed under both the MIT license and Apache 2.0 License.

👋 | Contributing

🎈 Thanks for your help improving the project! We are so happy to have you! We have a contributing guide to help you get involved in the rama project.

Contributions often come from people who already know what they want, be it a fix for a bug they encountered, or a feature that they are missing. Please do always make a ticket if one doesn't exist already.

It's possible however that you do not yet know what specifically to contribute, and yet want to help out. For that we thank you. You can take a look at the open issues, and in particular:

good first issue: issues that are good for those new to the rama codebase;
easy: issues that are seen as easy;
mentor available: issues for which we offer mentorship;
low prio: low prio issues that have no immediate pressure to be finished quick, great in case you want to help out but can only do with limited time to spare;

In general, any issue not assigned already is free to be picked up by anyone else. Please do communicate in the ticket if you are planning to pick it up, as to avoid multiple people trying to solve the same one.

💡 Some issues have a needs input label. These mean that the issue is not yet ready for development. First of all prior to starting working on an issue you should always look for alignment with the rama maintainers. However these needs input issues require also prior R&D work:

add and discuss missing knowledge or other things not clear;

figure out pros and cons of the solutions (as well as what if we choose to not not resolve the issue);

discuss and brainstorm on possible implementations, desire features, consequences, benefits, ...

Only once this R&D is complete and alignment is confirmed, shall the feature be started to be implemented.

Should you want to contribure this project but you do not yet know how to program in Rust, you could start learning Rust with as goal to contribute as soon as possible to rama by using "the Rust 101 Learning Guide" as your study companion. Glen can also be hired as a mentor or teacher to give you paid 1-on-1 lessons and other similar consultancy services. You can find his contact details at https://www.glendc.com/.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in rama by you, shall be licensed as both MIT and Apache 2.0, without any additional terms or conditions.

Acknowledgements

Special thanks goes to all involved in developing, maintaining and supporting the Rust programming language, the Tokio ecosystem and all other crates that we depend upon. This also includes Hyper and its ecosystem as without those projects Rama would not be. The core http module of rama is a specialised fork of hyper and use the underlying h2 and h3 crates as dependencies.

Extra credits also go to Axum, from which ideas and code were copied as its a project very much in line with the kind of software we want Rama to be, but for a different purpose. Our hats also go off to Tower, its inventors and all the people and creatures that help make it be every day. The service concept is derived from Tower and many of our layers are a Tower fork, adapted where required or desired.

An extra big shoutout goes also to the online communities surrounding and part of these ecosystems. They are a great place to hangout and always friendly and helpful. Thanks.

💖 | Sponsors

Rama is completely free, open-source software which needs lots of effort and time to develop and maintain.

You can become a regular financial contributor to Rama by paying for a monthly subscription at polar.sh/plabayo. In case you want a specific github issue to be resolved or like you can also fund issues via that platform. One time contributions are possible as well and greatly appreciated.

Alternatively you can also become a (monthly subscriber) sponsor or pay a one-time donation via Github Sponsors. For one-time donations you are also free to use "Buy me a Coffee" or "Paypal Donations" in case you are more at ease with any of these.

Sponsors help us continue to maintain and improve rama, as well as other Free and Open Source (FOSS) technology. It also helps us to create educational content such as https://github.com/plabayo/learn-rust-101, and other open source libraries such as https://github.com/plabayo/tokio-graceful and https://venndb.rs.

Next to the many unpaid developer hours we put in a project such as rama, we also have plenty of costs, such as services ranging from hosting to Docker, but also tooling for developers and automated processing. All these costs money.

Sponsors receive perks and depending on your regular contribution it also allows you to rely on us for support and consulting.

Finally, you can also support us by shopping Plabayo <3 ラマ merchandise 🛍️ at https://plabayo.threadless.com/.

Rama Sponsors

We would like to extend our thanks to the following sponsors for funding Rama (ラマ) development. If you are interested in becoming a sponsor, you can do so by becoming a sponsor. One time payments are accepted at GitHub as well as at "Buy me a Coffee". One-time and monthly financial contributions are also possible via Paypal, should you feel more at ease with that at "Paypal Donations".

If you wish to financially support us through other means you can best start a conversation with us by sending an email to [email protected].

Premium Partners

Rama (ラマ) is bundled with Http/Tls emulation data, gathered for all major platforms and browsers using real devices by BrowserStack. It does this automatically every day by using our public Fingerprinting service which is hosted together with a database on fly.io.

We are grateful to both sponsors for sponsering us these cloud resources.

Professional Services

🤝 Enterprise support, software customisations, integrations, professional support, consultancy and training are available upon request by sending an email to [email protected]. Or get an entireprise subscription at polar.sh/plabayo.

These type of contracts are another way for you to be able to support the project and at the same time get serviced for your own needs and purposes.

Rama is licensed as both MIT and Apache 2.0, as such you are free to use and modify the source code for any purposes, including commercial goals. That said, we would appreciate it if you would consider becoming a sponsor of the project if you do end up using it for commcercial reasons.

🌱 | Alternatives

While there are a handful of proxies written in Rust, there are only two other Rust frameworks specifically made for proxy purposes. All other proxy codebases are single purpose code bases, some even just for learning purposes. Or are actually generic http/web libraries/frameworks that facilitate proxy features as an extra.

Cloudflare has been working on a proxy service framework, named pingora, since a couple of years already, and on the 28th of February of 2024 they also open sourced it.

Rama is not for everyone, but we sure hope it is right for you. If not, consider giving pingora a try, it might very well be the next best thing for you.

Secondly, ByteDance has an open source proxy framework written in Rust to developer forward and reverse proxies alike, named g3proxy.

❓| FAQ

Available at https://ramaproxy.org/book/faq.html.

Name		Name	Last commit message	Last commit date
Latest commit History 1,793 Commits
.github		.github
benches		benches
docs		docs
examples		examples
fuzz		fuzz
rama-cli		rama-cli
rama-core		rama-core
rama-dns		rama-dns
rama-error		rama-error
rama-fp		rama-fp
rama-haproxy		rama-haproxy
rama-http-backend		rama-http-backend
rama-http-core		rama-http-core
rama-http-types		rama-http-types
rama-http		rama-http
rama-macros		rama-macros
rama-net		rama-net
rama-proxy		rama-proxy
rama-socks5		rama-socks5
rama-tcp		rama-tcp
rama-tls		rama-tls
rama-ua		rama-ua
rama-udp		rama-udp
rama-utils		rama-utils
src		src
supply-chain		supply-chain
test-files		test-files
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
.taplo.toml		.taplo.toml
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
FORK.md		FORK.md
LICENSE-APACHE		LICENSE-APACHE
LICENSE-MIT		LICENSE-MIT
README.md		README.md
SECURITY.md		SECURITY.md
_typos.toml		_typos.toml
build.rs		build.rs
deny.toml		deny.toml
justfile		justfile
lychee.toml		lychee.toml
rust-toolchain.toml		rust-toolchain.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Licenses found

Repository files navigation

⌨️ | `rama` binary

🧪 | Experimental

📣 | Rama Ecosystem

📦 | Rama Crates

🏢 | Proxy Examples

🌐 | Web Services

🌐 | Web Service Examples

🧑‍💻 | Http Clients

🧑‍💻 | Http Client Example

⛨ | Safety

🦀 | Compatibility

Minimum supported Rust version

🧭 | Roadmap

📰 | Media Appearances

💼 | License

👋 | Contributing

Contribution

Acknowledgements

💖 | Sponsors

Rama Sponsors

Premium Partners

Professional Services

🌱 | Alternatives

❓| FAQ

⭐ | Stargazers

About

Licenses found

Releases

Sponsor this project

Contributors 16

Languages

License

Licenses found

plabayo/rama

Folders and files

Latest commit

History

Repository files navigation

⌨️ | rama binary

🧪 | Experimental

📣 | Rama Ecosystem

📦 | Rama Crates

🏢 | Proxy Examples

🌐 | Web Services

🌐 | Web Service Examples

🧑‍💻 | Http Clients

🧑‍💻 | Http Client Example

⛨ | Safety

🦀 | Compatibility

Minimum supported Rust version

🧭 | Roadmap

📰 | Media Appearances

💼 | License

👋 | Contributing

Contribution

Acknowledgements

💖 | Sponsors

Rama Sponsors

Premium Partners

Professional Services

🌱 | Alternatives

❓| FAQ

⭐ | Stargazers

About

Topics

Resources

License

Licenses found

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

Sponsor this project

Contributors 16

Languages

⌨️ | `rama` binary