diff --git a/README.md b/README.md index 8acdf3df..d2172c4f 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,7 @@ Webcmd Cloud can run supported commands and browser sessions on hosted infrastru | Plugin | Description | Author | | --- | --- | --- | +| [`omnisearch`](./plugins/omnisearch/) | No-login research across Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, and Bluesky | [Rishet Mehra](https://github.com/Rishet11) | | [`pypi`](./plugins/pypi/) | Inspect public Python package metadata, downloads, and releases from PyPI | [Kemal Kaya](https://github.com/yoldaolmak) | | [`skyscanner`](./plugins/skyscanner/) | Skyscanner flight search commands for Webcmd | [Rishabh](https://github.com/rishabhraj36) | diff --git a/plugins/omnisearch/README.md b/plugins/omnisearch/README.md new file mode 100644 index 00000000..0601cdfb --- /dev/null +++ b/plugins/omnisearch/README.md @@ -0,0 +1,128 @@ +# 🔎 OmniSearch + +**OmniSearch is a research assistant that reads the internet for you.** + +Ask it what people on **Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, and Bluesky** are saying, asking, and publishing — and it brings back a clear answer with links. + +**You never need to touch the command line.** This tool is for your AI agent (Claude Code, Codex, Cursor, ChatGPT), not for you. Just tell your agent what you want to learn, and it handles the rest. + +--- + +## How to use OmniSearch (no coding needed) + +There are three steps — and you only do the first one yourself. + +**Step 1 — Install once (5 minutes).** You add OmniSearch to your AI assistant, like installing an app once. Your agent runs this for you: + +``` +webcmd plugin install github:agentrhq/webcmd/omnisearch +``` + +**Step 2 — Tell your agent what you want to know.** + +You don't type commands. You just talk. Say something like: + +> "Before I build this, find out what people actually think about it." + +Your agent recognizes OmniSearch and runs it for you. + +**Step 3 — Read the answer.** + +You get a plain-language summary — what people say, where they say it, and links to the actual posts. No folders, no files, nothing to interpret. + +--- + +## What you say. What your agent does. + +Here are real conversations, ready to copy-paste. Swap the words in **{braces}** for your topic. + +| You say | What your agent finds | +|---|---| +| "**{My idea}** — do people actually want this?" | What people say across Hacker News, GitHub, and developer communities | +| "Find what people complain about when using **{a tool}**." | The real problems people report on Stack Overflow and GitHub | +| "Is **{this idea}** already done? Who's winning?" | The competition and reception across platforms | +| "Is **{this idea}** even technically possible yet?" | What research says is proven vs. still speculative | +| "Before I launch **{product}**, what could kill it?" | The risks, past failures, and strongest arguments for it | +| "What's the reaction to **{a product / a launch / the news}**?" | What people are saying on Hacker News, Lobsters, and Bluesky right now | + +### Ready-to-paste prompts for your agent + +Pick one, replace **{the brackets}**, and paste it to your agent: + +1. "Research **{topic/competitor/product}**. Use OmniSearch to check Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, and Lobsters. Give me a two-paragraph summary and the 5 most-talked-about results with links." + +2. "Find what's broken or frustrating about **{topic}**. Look at Stack Overflow questions and GitHub issues, and list the 10 most common problems people mention, each with a source link." + +3. "Check if **{my idea}** is technically realistic yet. Scan recent research papers and tech discussions, and tell me what's proven to work vs. still speculative." + +4. "Validate my idea: **{one-line pitch}**. Gather what people are saying about it, rate how much real interest there is, and quote 3 real people with links." + +5. "Do a pre-mortem on **{my startup/product}**. Find how similar things failed before, the top 3 reasons this could fail, and the strongest argument it could succeed — each with a source." + +6. "What do developers complain about when using **{a tool/language}**? Summarize the biggest recurring pain points with links." + +7. "Check the reaction to **{a new release / launch}**. What are people on Hacker News, Lobsters, and Bluesky saying right now?" + +--- + +## What you can research + +OmniSearch reads **7 public sources** without requiring an account: + +| Platform | Best at telling you | +|---|---| +| **Hacker News** 🟠 | what the tech world thinks | +| **Stack Overflow** 📚 | what's actually broken | +| **GitHub** 🐙 | real problems people file | +| **Dev.to** 💜 | what developers are writing | +| **arXiv** 📄 | the latest research | +| **Lobsters** 🦞 | smart developer discussion | +| **Bluesky** 🦋 | fresh public posts | + +**No login is needed.** Each command uses a public API or feed. + +--- + +## Start now + +**Install OmniSearch once, then ask your agent:** + +> *"Before I build this, find out what people actually think about it."* + +That's the whole product. + +--- + +## For developers + +Everything below is for the agent (and for people who want to dig in). A normal user never reads this. + +### The commands + +```bash +# Aggregate everything about a topic +webcmd omnisearch research "" -f json + +# 🏆 The community's verdict, distilled +webcmd omnisearch verdict "" -f json + +# One source at a time +webcmd omnisearch hackermind "" -f json +webcmd omnisearch stackoverflow "" -f json +webcmd omnisearch github "" -f json +webcmd omnisearch arxiv "" -f json +webcmd omnisearch research --sources devto -f json +webcmd omnisearch lobsters -f json +webcmd omnisearch bluesky-posts -f json + +``` + +### Output + +The research and source-search commands return `platform`, `title`, `author`, `score`, `commentCount`, `createdAt`, `url`, and `text`. Feed commands expose source-specific fields. + +### How it works + +The aggregated research sources live in `sources.js` behind one signature (`search(query, limit)`). + +Ships `SKILL.md` so agent harnesses auto-discover OmniSearch. diff --git a/plugins/omnisearch/SKILL.md b/plugins/omnisearch/SKILL.md new file mode 100644 index 00000000..a02cb877 --- /dev/null +++ b/plugins/omnisearch/SKILL.md @@ -0,0 +1,26 @@ +--- +name: omnisearch +description: Read-only, no-login web research across Hacker News, Lobste.rs, Stack + Overflow, GitHub issues, Dev.to, arXiv, and Bluesky. Use to answer "what does the + internet think / ask / struggle with" about any topic. Returns JSON via webcmd. +--- + +# OmniSearch skill + +1. Invoke every query as `webcmd omnisearch -f json`. The site name is + `omnisearch` — never call `webcmd research` or omit the site prefix. +2. Choose the command by intent: + - `research ""` — whole-space scan across all sources. Add + `--sources hn,lobsters,stackoverflow,github,arxiv` to narrow. + - `github` / `stackoverflow` — real problems people report/ask. + - `hackermind ""` — HN opinions and discussions. + - `arxiv ""` — research papers. + - `research --sources devto` — Dev.to articles; use a tag such as `saas`. + - `lobsters --sort active|newest|hot` — live discussion (no free query). + - `bluesky-posts ` — one public account's feed. +3. `research` distributes `--limit` across sources, so use `--limit` >= 24 for + coverage. `title` and `url` are always populated; `text` may be empty (HN, + StackOverflow) or truncated (GitHub 300 chars, arXiv 200). Never fabricate a + summary from an empty `text` — cite `url` and rank by `score`. +4. GitHub search is rate-limited (~10 req/min). Space out parallel calls; do not + fan out 6 GitHub calls at once. diff --git a/plugins/omnisearch/arxiv.js b/plugins/omnisearch/arxiv.js new file mode 100644 index 00000000..f7487b49 --- /dev/null +++ b/plugins/omnisearch/arxiv.js @@ -0,0 +1,43 @@ +/** + * OmniSearch arxiv — search arXiv research papers. + * No login. Uses the public arXiv Atom API. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { arxivSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'arxiv', + tags: ['search'], + access: 'read', + description: "Search arXiv research papers (no login)", + domain: 'export.arxiv.org', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Research topic' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await arxivSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/arxiv', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/bluesky-posts.js b/plugins/omnisearch/bluesky-posts.js new file mode 100644 index 00000000..06232ea7 --- /dev/null +++ b/plugins/omnisearch/bluesky-posts.js @@ -0,0 +1,89 @@ +/** + * omnisearch bluesky-posts — read what a public Bluesky account is saying. + * + * No login. Uses the public Bluesky API (public.api.bsky.app) author feed + * endpoint, which returns a user's recent posts without authentication. + * This is the closest no-login analog to reading someone's public X/Twitter + * timeline. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { + ArgumentError, + CommandExecutionError, + EmptyResultError, +} from '@agentrhq/webcmd/errors'; + +const API = 'https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed'; +const HANDLE = /^[a-zA-Z0-9][a-zA-Z0-9.-]*\.[a-zA-Z]{2,}$/; + +function requireHandle(value) { + const s = String(value ?? '').trim().toLowerCase(); + if (!s || !HANDLE.test(s)) { + throw new ArgumentError('bluesky handle is required, e.g. "bsky.app" or "user.bsky.social"'); + } + return s; +} + +cli({ + site: 'omnisearch', + name: 'bluesky-posts', + access: 'read', + description: "Recent posts from a public Bluesky account (no login)", + domain: 'public.api.bsky.app', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'handle', required: true, positional: true, help: 'Bluesky handle (e.g. bsky.app)' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of posts' }, + ], + columns: ['rank', 'uri', 'createdAt', 'text', 'likeCount', 'replyCount', 'repostCount', 'url'], + func: async (kwargs) => { + const handle = requireHandle(kwargs.handle); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + + const url = new URL(API); + url.searchParams.set('actor', handle); + url.searchParams.set('limit', String(limit)); + + let json; + try { + const res = await fetch(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; webcmd-omnisearch/0.1)' }, + }); + if (!res.ok) { + throw new CommandExecutionError(`Bluesky API request failed: HTTP ${res.status}`); + } + json = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`Bluesky API request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const feed = Array.isArray(json?.feed) ? json.feed : []; + if (!feed.length) { + throw new EmptyResultError('omnisearch/bluesky-posts', `no posts found for "${handle}"`); + } + + return feed.slice(0, limit).map((entry, index) => { + const post = entry?.post ?? {}; + const author = post.author ?? {}; + const record = post.record ?? {}; + const uri = String(post.uri ?? ''); + const rkey = uri.split('/').pop() ?? ''; + return { + rank: index + 1, + uri, + createdAt: String(record.createdAt ?? post.indexedAt ?? ''), + text: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim(), + likeCount: post.likeCount ?? 0, + replyCount: post.replyCount ?? 0, + repostCount: post.repostCount ?? 0, + url: `https://bsky.app/profile/${author.handle ?? handle}/post/${rkey}`, + }; + }); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/github.js b/plugins/omnisearch/github.js new file mode 100644 index 00000000..acb9f930 --- /dev/null +++ b/plugins/omnisearch/github.js @@ -0,0 +1,43 @@ +/** + * OmniSearch github — search GitHub issues & PRs (people reporting real problems). + * No login. Uses the public GitHub search API (rate-limited to ~10/min). + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { githubSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'github', + tags: ['search'], + access: 'read', + description: "Search GitHub issues & PRs for real problems (no login)", + domain: 'api.github.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Problem or feature to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await githubSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/github', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/hackermind.js b/plugins/omnisearch/hackermind.js new file mode 100644 index 00000000..ff92ab65 --- /dev/null +++ b/plugins/omnisearch/hackermind.js @@ -0,0 +1,80 @@ +/** + * omnisearch hackermind — search Hacker News stories & comments for omnisearch. + * + * No login. Uses the public HN Algolia API, which indexes stories and + * comments. Good for researching what the tech/startup community is saying + * about a topic, product, or problem. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; + +const API = 'https://hn.algolia.com/api/v1/search'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'hackermind', + tags: ['search'], + access: 'read', + description: "Search Hacker News stories & comments (no login)", + domain: 'hn.algolia.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Topic, product, or problem to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + { + name: 'scope', + default: 'story', + help: 'What to search: story (headlines) or comment (reply text)', + choices: ['story', 'comment'], + }, + ], + columns: ['rank', 'id', 'objectType', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + const scope = String(kwargs.scope ?? 'story'); + + const url = new URL(API); + url.searchParams.set('query', query); + url.searchParams.set('tags', scope === 'comment' ? 'comment' : 'story'); + url.searchParams.set('hitsPerPage', String(limit)); + + let json; + try { + const res = await fetch(url); + if (!res.ok) throw new CommandExecutionError(`HN Algolia request failed: HTTP ${res.status}`); + json = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`HN Algolia request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const hits = Array.isArray(json?.hits) ? json.hits : []; + if (!hits.length) { + throw new EmptyResultError('omnisearch/hackermind', `no results for "${query}"`); + } + + return hits.slice(0, limit).map((h, index) => ({ + rank: index + 1, + id: h.objectID, + objectType: scope, + title: String(h.title ?? h.story_title ?? h.comment_text ?? '').replace(/<[^>]+>/g, '').trim(), + author: String(h.author ?? ''), + score: h.points ?? 0, + commentCount: h.num_comments ?? 0, + createdAt: String(h.created_at ?? ''), + url: String(h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`), + })); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/lobsters.js b/plugins/omnisearch/lobsters.js new file mode 100644 index 00000000..c7933329 --- /dev/null +++ b/plugins/omnisearch/lobsters.js @@ -0,0 +1,73 @@ +/** + * omnisearch lobsters — Lobste.rs newest / active discussions. + * + * No login. Uses the public lobste.rs JSON endpoint. Lobste.rs is a + * Reddit/HN-style community; good for surfacing what developers are + * discussing about a topic right now. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; + +const SORTS = { + newest: 'https://lobste.rs/newest.json', + active: 'https://lobste.rs/active.json', + hot: 'https://lobste.rs/hottest.json', +}; + +function requireSort(value) { + const s = String(value ?? 'newest').toLowerCase(); + if (s === 'new') s = 'newest'; + if (!SORTS[s]) throw new ArgumentError(`sort must be one of: ${Object.keys(SORTS).join(', ')}`); + return s; +} + +cli({ + site: 'omnisearch', + name: 'lobsters', + access: 'read', + description: "Lobste.rs newest / active / hot discussions (no login)", + domain: 'lobste.rs', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'limit', type: 'int', default: 20, help: 'Number of stories' }, + { name: 'sort', default: 'newest', help: 'Sort order: newest, active, hot', choices: ['newest', 'active', 'hot'] }, + ], + columns: ['rank', 'id', 'title', 'author', 'score', 'commentCount', 'createdAt', 'tags', 'url'], + func: async (kwargs) => { + const sort = requireSort(kwargs.sort); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + + let rows; + try { + const res = await fetch(SORTS[sort], { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; webcmd-omnisearch/0.1)' }, + }); + if (!res.ok) throw new CommandExecutionError(`lobste.rs request failed: HTTP ${res.status}`); + rows = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`lobste.rs request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + if (!Array.isArray(rows) || !rows.length) { + throw new EmptyResultError('omnisearch/lobsters', 'lobste.rs returned no stories'); + } + + return rows.slice(0, limit).map((s, index) => ({ + rank: index + 1, + id: s.short_id, + title: String(s.title ?? '').trim(), + author: String(s.submitter_user ?? ''), + score: s.score ?? 0, + commentCount: s.comment_count ?? 0, + createdAt: String(s.created_at ?? ''), + tags: Array.isArray(s.tags) ? s.tags.join(', ') : '', + url: String(s.comments_url ?? s.short_id_url ?? ''), + })); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/package.json b/plugins/omnisearch/package.json new file mode 100644 index 00000000..803ff4d6 --- /dev/null +++ b/plugins/omnisearch/package.json @@ -0,0 +1,9 @@ +{ + "name": "webcmd-plugin-omnisearch", + "version": "0.1.0", + "type": "module", + "description": "No-login research across seven public platforms", + "peerDependencies": { + "@agentrhq/webcmd": ">=0.5.3" + } +} diff --git a/plugins/omnisearch/research.js b/plugins/omnisearch/research.js new file mode 100644 index 00000000..ef636f5f --- /dev/null +++ b/plugins/omnisearch/research.js @@ -0,0 +1,86 @@ +/** + * OmniSearch research — aggregate results about a topic across all public + * platforms into one feed. + * + * No login. Combines Hacker News, Lobste.rs, Stack Overflow, Dev.to, GitHub + * issues, and arXiv, tagging each row with its platform. Use for quick + * universal reconnaissance on a topic, product, or problem. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { + hnSearch, + lobstersSearch, + stackoverflowSearch, + devtoSearch, + githubSearch, + arxivSearch, +} from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a research topic/query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'research', + tags: ['search'], + access: 'read', + description: "Aggregate results about a topic across all public platforms (Hacker News, Lobsters, Stack Overflow, Dev.to, GitHub, arXiv)", + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Topic, product, or problem to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Maximum total results' }, + { + name: 'sources', + default: 'hn,lobsters,stackoverflow,devto,github,arxiv', + help: 'Comma-separated sources to query (default: all)', + }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 50); + + const wanted = String(kwargs.sources ?? '') + .split(',') + .map((s) => s.trim().toLowerCase()) + .filter(Boolean); + + const fetchers = { + hn: () => hnSearch(query, perPlatform), + lobsters: () => lobstersSearch(query, perPlatform), + stackoverflow: () => stackoverflowSearch(query, perPlatform), + devto: () => devtoSearch(query, perPlatform), + github: () => githubSearch(query, perPlatform), + arxiv: () => arxivSearch(query, perPlatform), + }; + + const selected = wanted.length ? wanted.filter((s) => fetchers[s]) : Object.keys(fetchers); + const perPlatform = Math.ceil(limit / Math.max(selected.length, 1)); + + let rows; + try { + // Failure isolation: one rate-limited/erroring source must not wipe out the rest. + const outcomes = await Promise.allSettled(selected.map((key) => fetchers[key]())); + rows = outcomes + .filter((o) => o.status === 'fulfilled') + .flatMap((o) => o.value); + } catch (err) { + throw new CommandExecutionError(`research aggregation failed: ${err instanceof Error ? err.message : String(err)}`); + } + + if (!rows.length) { + throw new EmptyResultError('omnisearch/research', `no results found across platforms for "${query}"`); + } + + return rows.slice(0, limit); + }, +}); diff --git a/plugins/omnisearch/sources.js b/plugins/omnisearch/sources.js new file mode 100644 index 00000000..5fedf25a --- /dev/null +++ b/plugins/omnisearch/sources.js @@ -0,0 +1,194 @@ +/** + * OmniSearch — shared source fetchers for public platforms. + * + * Every search command aggregates across these. Each fetcher returns + * normalized rows: { platform, title, author, score, commentCount, createdAt, url, text } + */ +import { CommandExecutionError } from '@agentrhq/webcmd/errors'; + +/** Fetch helper: isolates transport errors into webcmd's typed error. */ +async function get(url, init, { source } = {}) { + let res; + try { + res = await fetch(url, init); + } catch (err) { + throw new CommandExecutionError( + `OmniSearch: ${source} request failed: ${err instanceof Error ? err.message : String(err)}`, + ); + } + if (!res.ok) { + throw new CommandExecutionError(`OmniSearch: ${source} HTTP ${res.status}`); + } + return res; +} + +// --- Hacker News (Algolia) --- +export async function hnSearch(query, limit) { + const url = new URL('https://hn.algolia.com/api/v1/search'); + url.searchParams.set('query', query); + url.searchParams.set('tags', 'story'); + url.searchParams.set('hitsPerPage', String(limit)); + const res = await get(url, {}, { source: 'Hacker News' }); + const json = await res.json(); + return (json?.hits ?? []).slice(0, limit).map((h) => ({ + platform: 'hackernews', + title: String(h.title ?? h.story_title ?? '').trim(), + author: String(h.author ?? ''), + score: h.points ?? 0, + commentCount: h.num_comments ?? 0, + createdAt: String(h.created_at ?? ''), + url: String(h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`), + text: '', + })); +} + +// --- Lobste.rs --- +export async function lobstersSearch(query, limit) { + const res = await get('https://lobste.rs/newest.json', { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Lobste.rs' }); + const rows = await res.json(); + if (!Array.isArray(rows)) return []; + const q = query.toLowerCase(); + return rows + .filter((s) => + String(s.title ?? '').toLowerCase().includes(q) || + String(s.description_plain ?? '').toLowerCase().includes(q) || + (Array.isArray(s.tags) && s.tags.some((t) => t.toLowerCase().includes(q))), + ) + .slice(0, limit) + .map((s) => ({ + platform: 'lobsters', + title: String(s.title ?? '').trim(), + author: String(s.submitter_user ?? ''), + score: s.score ?? 0, + commentCount: s.comment_count ?? 0, + createdAt: String(s.created_at ?? ''), + url: String(s.comments_url ?? ''), + text: String(s.description_plain ?? ''), + })); +} + +// --- Stack Overflow (public StackExchange API) --- +export async function stackoverflowSearch(query, limit) { + const url = new URL('https://api.stackexchange.com/2.3/search/advanced'); + url.searchParams.set('order', 'desc'); + url.searchParams.set('sort', 'relevance'); + url.searchParams.set('q', query); + url.searchParams.set('site', 'stackoverflow'); + url.searchParams.set('pagesize', String(limit)); + const res = await get(url, {}, { source: 'Stack Overflow' }); + const json = await res.json(); + return (json?.items ?? []).slice(0, limit).map((q) => ({ + platform: 'stackoverflow', + title: String(q.title ?? '').trim(), + author: String(q.owner?.display_name ?? ''), + score: q.score ?? 0, + commentCount: q.answer_count ?? 0, + createdAt: String(new Date((q.creation_date ?? Date.now()) * 1000).toISOString()), + url: String(q.link ?? ''), + text: '', + })); +} + +// --- Dev.to (tag-based public articles) --- +export async function devtoSearch(query, limit) { + const url = new URL('https://dev.to/api/articles'); + url.searchParams.set('tag', query.replace(/[^a-zA-Z0-9-]/g, '')); + url.searchParams.set('per_page', String(limit)); + const res = await get(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Dev.to' }); + const json = await res.json(); + if (!Array.isArray(json)) return []; + return json.slice(0, limit).map((a) => ({ + platform: 'devto', + title: String(a.title ?? '').trim(), + author: String(a.user?.username ?? a.user?.name ?? ''), + score: a.positive_reactions_count ?? 0, + commentCount: a.comments_count ?? 0, + createdAt: String(a.published_at ?? ''), + url: String(a.url ?? ''), + text: String(a.description ?? ''), + })); +} +// --- GitHub issues (people reporting problems) --- +export async function githubSearch(query, limit) { + const url = new URL('https://api.github.com/search/issues'); + url.searchParams.set('q', `${query} in:title,body`); + url.searchParams.set('per_page', String(limit)); + const res = await get(url, { headers: { 'User-Agent': 'OmniSearch/0.1' } }, { source: 'GitHub' }); + const json = await res.json(); + return (json?.items ?? []).slice(0, limit).map((i) => ({ + platform: 'github', + title: `[${i.state ?? ''}] ${String(i.title ?? '').trim()}`.trim(), + author: String(i.user?.login ?? ''), + score: i.reactions?.total_count ?? 0, + commentCount: i.comments ?? 0, + createdAt: String(i.created_at ?? ''), + url: String(i.html_url ?? ''), + text: String(i.body ?? '').slice(0, 300), + })); +} + +// --- arXiv (research papers) --- +export async function arxivSearch(query, limit) { + const url = new URL('https://export.arxiv.org/api/query'); + url.searchParams.set('search_query', `all:${query.split(' ').join('+')}`); + url.searchParams.set('max_results', String(limit)); + const res = await get(url, {}, { source: 'arXiv' }); + const xml = await res.text(); + const entryRe = /([\s\S]*?)<\/entry>/g; + const rows = []; + let m; + while ((m = entryRe.exec(xml)) !== null && rows.length < limit) { + const e = m[1]; + const title = (e.match(/([\s\S]*?)<\/title>/) || [])[1] + ?.replace(/\s+/g, ' ').trim() ?? ''; + const author = ((e.match(/<name>([\s\S]*?)<\/name>/) || [])[1] ?? '').trim(); + const id = (e.match(/<id>([\s\S]*?)<\/id>/) || [])[1] ?? ''; + const updated = ((e.match(/<updated>([\s\S]*?)<\/updated>/) || [])[1] ?? '').trim(); + const summary = ((e.match(/<summary>([\s\S]*?)<\/summary>/) || [])[1] ?? '') + .replace(/\s+/g, ' ').trim(); + rows.push({ + platform: 'arxiv', + title, + author, + score: 0, + commentCount: 0, + createdAt: updated, + url: id, + text: summary.slice(0, 200), + }); + } + return rows; +} + +// --- Bluesky (public author feed) --- +export async function blueskyPosts(handle, limit) { + const url = new URL('https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed'); + url.searchParams.set('actor', handle); + url.searchParams.set('limit', String(limit)); + const res = await get(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Bluesky' }); + const json = await res.json(); + const feed = Array.isArray(json?.feed) ? json.feed : []; + return feed.slice(0, limit).map((entry) => { + const post = entry?.post ?? {}; + const author = post.author ?? {}; + const record = post.record ?? {}; + const uri = String(post.uri ?? ''); + const rkey = uri.split('/').pop() ?? ''; + return { + platform: 'bluesky', + title: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim().slice(0, 200), + author: String(author.handle ?? handle), + score: post.likeCount ?? 0, + commentCount: post.replyCount ?? 0, + createdAt: String(record.createdAt ?? post.indexedAt ?? ''), + url: `https://bsky.app/profile/${author.handle ?? handle}/post/${rkey}`, + text: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim(), + }; + }); +} diff --git a/plugins/omnisearch/stackoverflow.js b/plugins/omnisearch/stackoverflow.js new file mode 100644 index 00000000..60c082b6 --- /dev/null +++ b/plugins/omnisearch/stackoverflow.js @@ -0,0 +1,43 @@ +/** + * OmniSearch stackoverflow — search Stack Overflow for real problems & questions. + * No login. Uses the public StackExchange API. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { stackoverflowSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'stackoverflow', + tags: ['search'], + access: 'read', + description: "Search Stack Overflow questions & problems (no login)", + domain: 'api.stackexchange.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Problem or question to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await stackoverflowSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/stackoverflow', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/test/research.test.js b/plugins/omnisearch/test/research.test.js new file mode 100644 index 00000000..699fe6cb --- /dev/null +++ b/plugins/omnisearch/test/research.test.js @@ -0,0 +1,29 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { getRegistry } from '@agentrhq/webcmd/registry'; +import '../research.js'; + +afterEach(() => vi.unstubAllGlobals()); + +describe('omnisearch research', () => { + it('honors the total limit when research is narrowed to one source', async () => { + const hits = [ + { objectID: '1', title: 'One', author: 'a', points: 5, num_comments: 1, created_at: '2026-01-01T00:00:00Z', url: 'https://example.com/1' }, + { objectID: '2', title: 'Two', author: 'b', points: 4, num_comments: 2, created_at: '2026-01-02T00:00:00Z', url: 'https://example.com/2' }, + { objectID: '3', title: 'Three', author: 'c', points: 3, num_comments: 3, created_at: '2026-01-03T00:00:00Z', url: 'https://example.com/3' }, + { objectID: '4', title: 'Four', author: 'd', points: 2, num_comments: 4, created_at: '2026-01-04T00:00:00Z', url: 'https://example.com/4' }, + { objectID: '5', title: 'Five', author: 'e', points: 1, num_comments: 5, created_at: '2026-01-05T00:00:00Z', url: 'https://example.com/5' }, + ]; + vi.stubGlobal('fetch', async (input) => { + const count = Number(new URL(input).searchParams.get('hitsPerPage')); + return new Response(JSON.stringify({ hits: hits.slice(0, count) }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }); + }); + const command = getRegistry().get('omnisearch/research'); + + const rows = await command.func({ query: 'webcmd', limit: 5, sources: 'hn' }); + + expect(rows.map((row) => row.title)).toEqual(['One', 'Two', 'Three', 'Four', 'Five']); + }); +}); diff --git a/plugins/omnisearch/verdict.js b/plugins/omnisearch/verdict.js new file mode 100644 index 00000000..b37b7a06 --- /dev/null +++ b/plugins/omnisearch/verdict.js @@ -0,0 +1,79 @@ +/** + * OmniSearch verdict — synthesize what the community thinks about a topic. + * + * Runs the multi-source research sweep, then returns an opinionated summary: + * per-platform top results + the single highest-traction item + which platforms + * are most engaged. This is "signal, not search" — a verdict, not a link dump. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { hnSearch, lobstersSearch, stackoverflowSearch, githubSearch, arxivSearch, devtoSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a topic is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'verdict', + tags: ['search'], + access: 'read', + description: "Synthesize the community's verdict on a topic across all public platforms (signal, not search)", + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'topic', required: true, positional: true, help: 'Topic to synthesize' }, + { name: 'perSource', type: 'int', default: 3, help: 'Top results per source to consider' }, + ], + columns: ['verdict', 'topResult', 'topSource', 'topScore', 'platforms', 'totalResults'], + func: async (kwargs) => { + const topic = requireQuery(kwargs.topic); + const raw = Number(kwargs.perSource ?? 3); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('perSource must be a positive integer'); + const perSource = Math.min(raw, 10); + + const fetchers = [ + () => hnSearch(topic, perSource), + () => stackoverflowSearch(topic, perSource), + () => githubSearch(topic, perSource), + () => arxivSearch(topic, perSource), + () => devtoSearch(topic, perSource), + () => lobstersSearch(topic, perSource), + ]; + + let results; + try { + const outcomes = await Promise.allSettled(fetchers.map((f) => f())); + results = outcomes.filter((o) => o.status === 'fulfilled').map((o) => o.value); + } catch (err) { + throw new CommandExecutionError(`verdict failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const all = results.flat(); + if (!all.length) { + throw new EmptyResultError('omnisearch/verdict', `no results for "${topic}"`); + } + + // Highest-traction single result across all sources (by score, fallback commentCount) + const top = all.reduce((a, b) => { + const as = (a.score ?? 0) + (a.commentCount ?? 0); + const bs = (b.score ?? 0) + (b.commentCount ?? 0); + return bs > as ? b : a; + }); + + const platformCounts = {}; + for (const r of all) platformCounts[r.platform] = (platformCounts[r.platform] ?? 0) + 1; + const platforms = Object.entries(platformCounts).map(([p, n]) => `${p}(${n})`).join(', '); + + return [{ + verdict: `Community signal on "${topic}": strongest result is "${top.title.slice(0, 100)}" on ${top.platform} (score ${top.score}, ${top.commentCount} comments).`, + topResult: top.title, + topSource: top.platform, + topScore: (top.score ?? 0) + (top.commentCount ?? 0), + platforms, + totalResults: all.length, + }]; + }, +}); diff --git a/plugins/omnisearch/webcmd-plugin.json b/plugins/omnisearch/webcmd-plugin.json new file mode 100644 index 00000000..671a4f00 --- /dev/null +++ b/plugins/omnisearch/webcmd-plugin.json @@ -0,0 +1,10 @@ +{ + "name": "omnisearch", + "version": "0.1.0", + "description": "No-login research across Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, and Bluesky", + "webcmd": ">=0.5.3", + "author": { + "name": "Rishet Mehra", + "handle": "Rishet11" + } +} diff --git a/webcmd-plugin.json b/webcmd-plugin.json index 3170090d..9d38e18d 100644 --- a/webcmd-plugin.json +++ b/webcmd-plugin.json @@ -774,6 +774,16 @@ "handle": "agentrhq" } }, + "omnisearch": { + "path": "plugins/omnisearch", + "version": "0.1.0", + "description": "No-login research across Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, and Bluesky", + "webcmd": ">=0.5.3", + "author": { + "name": "Rishet Mehra", + "handle": "Rishet11" + } + }, "openalex": { "path": "plugins/openalex", "version": "0.1.0",