gosquito

gosquito ("go" + "mosquito") is a pluggable tool for data gathering, data processing and data transmitting to various destinations. Main goal is to replace various in-house automated tasks with a single tool and move those tasks at the edge. See docs and examples for additional info.

graph TD;
    I1(io)-->P1(process plugins);
    I2(kafka)-->P1;
    I3(resty)-->P1;
    I4(rss)-->P1;
    I5(telegram)-->P1;
    P1-->O1(kafka);
    P1-->O2(mattermost);
    P1-->O3(resty);
    P1-->O4(slack);
    P1-->O5(smtp);
    P1-->O6(telegram);

Loading

Main features:

• Pluggable architecture. Data processing organized as chains of plugins.
• Flow approach. Flow consists of: input plugin (receive), process plugins (filter/transform), output plugin (send).
• Declarative YAML configurations with templates support.
• Consider data as new by configurable signature (data timestamp by default). Force fetching is supported.
• Dependencies between process plugins. Plugin "B" will process data only if plugin "A" derived some data.
• Parallel executions limits.
• Export metrics to Prometheus.

Input plugins:

Plugin	Description
io	Use text and files as data source.
kafka	Kafka topic as data source.
resty	REST endpoint as data source.
rss	RSS/Atom feed as data source.
telegram	Telegram chat as data source.

Process plugins:

Plugin	Description
dedup	Deduplicate datums by UUID.
echo	Echoing processing data.
expandurl	Expand short URLs.
fetch	Fetch remote data.
iconv	Convert text encoding.
io	Read/write text and files.
jq	Extract JSON elements.
minio	Get/put data from/to S3 bucket.
regexpfind	Find patters in data.
regexpmatch	Match data by patterns.
regexpreplace	Replace patterns in data.
resty	Perform REST queries.
same	Match data similarity.
split	Split single datum into multiple.
unique	Get unique values from data.
webchela	Process web pages with full browser support (Chrome, Firefox).
xpath	Extract HTML/XML nodes.

Output plugins:

Plugin	Description
kafka	Send data to Kafka topic.
mattermost	Send data to Mattermost channel/user.
resty	Send data to REST endpoint.
slack	Send data to Slack channel/user.
smtp	Send data as email.
telegram	Send data to Telegram chat.