Skip to content

Commit

Permalink
feat: add get_web_page (#11)
Browse files Browse the repository at this point in the history
  • Loading branch information
David-Else committed Jun 1, 2024
1 parent 55a72d7 commit 842f5f0
Showing 1 changed file with 14 additions and 0 deletions.
14 changes: 14 additions & 0 deletions sh/get_web_page.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -e

# @describe Fetches the HTML content from a specified webpage URL and converts it to Markdown format.
# Use it to answer user questions that require up-to-date content from web pages.
# @meta require-tools curl,pandoc,sed
# @option --url! The URL to scrape.

main() {
# span and div tags are dropped from the HTML https://pandoc.org/MANUAL.html#raw-htmltex and sed removes any inline SVG images in image tags from the Markdown content.
curl -fsSL "$argc_url" | pandoc -f html-native_divs-native_spans -t gfm-raw_html | sed -E 's/!\[.*?\]\((data:image\/svg\+xml[^)]+)\)//g'
}

eval "$(argc --argc-eval "$0" "$@")"

0 comments on commit 842f5f0

Please sign in to comment.