Blogging in Gatsby with Drummer

[scotthansonde]

I'm now able to use Drummer to post to my blog in Gatsby! I described it in a post that of course is written in Drummer.

My code is at https://github.com/papascott/papascott-gatsby

I started with the gatsby-demo-opmlnotes by Andrew Shell

[scripting]

Congrats and good work! 😀

A suggestion, link to Gatsby and Drummer the first time their names appear in the post.

I’m not sure about Gatsby, would want to know more, and I imagine most Gatsby users don’t know about Drummer and what it means for it to work with Gatsby.

[scotthansonde]

I've added the links, thanks for the suggestion. In a future post I can explain a little bit about Gatsby and why I use it. Right now I'm still at the point where I'm astounded that my code seems to work as I intended! 😃

[scripting]

@papascott -- I published a writeup on Scripting News.

I'm banging the drum for connecting stuff together. And this is an important milestone.

Maybe a video?

Is there a quick tutorial for Gatsby that assumes I understand HTML, have Node installed, have run CMSes.

I looked at their tutorial and simply can't find out what it;'s like to use Gatsby so I can't see how Drummer connects.

[scotthansonde]

@scripting Yeah, the present Gatsby site is quite startup-intense. A better technical explanation might be their GitHub Readme or this 13 minute video talk on What Is Gatsby. I also found a 45 minute video tutorial on Learn Gatsby: How to Build Your First Gatsby Blog where they start with a "blog starter" to create an ugly but functional blog.

My executive summary:

• "Gatsby is a free and open source framework based on React", so the templates are written in React.
• It renders static files, so it can be served at any service that can serve static files (like S3 or Netlify).
• And relevant to my solution, it can "Load Data From Anywhere". It uses "source plugins" to pull in data from markdown files, a CMS like WordPress, or an API. The data, regardless of source, is exposed on a GraphQL API; that's how the data gets in the templates.

Andrew Shell wrote a source plugin for Little Outliner OPML files. I was able to use that with minor modifications to import Drummer OPML data into my blog.

I appreciated your remark about plumbers and poets in the same space. I'm certainly a plumber! 😄

[scripting]

@papascott -- glad you saw plumbers and poets. it just came to me that's what we are. i am a plumber for sure. i even have a friend who said i was the best plumber ever. i really appreciated that. ;-)

you know the video i want to see -- you, using drummer to publish to gatsby. do a seat of the pants demo in youtube, that would give me most of the info i need. that's the level at which gatsby is relevant to me.

[scotthansonde]

I’ll be happy to make that video, either tomorrow or Wednesday! 😀

…

On 1. Nov 2021, at 20:10, Dave Winer ***@***.***> wrote:  @papascott -- glad you saw plumbers and poets. it just came to me that's what we are. i am a plumber for sure. i even have a friend who said i was the best plumber ever. i really appreciated that. ;-) you know the video i want to see -- you, using drummer to publish to gatsby. do a seat of the pants demo in youtube, that would give me most of the info i need. that's the level at which gatsby is relevant to me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[scotthansonde]

I've recorded a screencast of making a post: https://youtu.be/9Y4cn-GPamY

[thejeshgn]

@papascott it leaks your netlify webhook URL. Hopefully the URL is inactive now.

[scotthansonde]

@thejeshgn No worries, there's a new URL now 😄

[kevinctofel]

@papascott this is a great video demo! I’ve been using Gatsby for my personal blog for about 18 months now and you nailed the experience.

@scripting I’d say the “integration pipes” between Drummer and Gatsby are already there via GitHub and Netlify. Netlify builds and publishes static site files by reading them in .md format from GitHub. So if a tool can be used to save blog posts during creation to GitHub as .md, (which is what @papascott’s webhook does, as soon as Netlify sees a change to the GitHub repo, it regenerates and publishes the blog post pages. High level, but just some additional context on the process of Gatsby/Netlify.

My first explanation of Gatsby when I decided to use it is here. So as long as something can modify Drummer blog posts to a GitHub repo in .md that Netlify is configured to watch, Netlify will use Gatsby to publish those posts by default.

[scripting]

@papascott — okay now I understand what the pieces are, and I think I understand what your drummer script is doing, but the text is too small to read. Are you sending the whole opml file over to Gatsby? Is your plug-in running on Gatsby, parsing the opml and converting it to a format Gatsby understands (presumably markdown?)

@kevinctofel - great to see you here! Drummer is not writing anything to GitHub, so whatever Scott is doing, he’s not depending on that.

anyway you both have given me lots to think about! 😀

[scotthansonde]

@scripting Here's my Drummer script. It in fact sends an empty object to Netlify. All hitting the URL does is trigger the build.

const xhr = new XMLHttpRequest();
const hookUrl="https://api.netlify.com/build_hooks/XXXXXXX"
xhr.open("POST", hookUrl, true);
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.send(JSON.stringify({}));
dialog.alert ('Sent build hook to Netlify')

The plugin itself fetches the outline from the public URL when the site is built. Here are the options I'm giving the plugin:

{
      resolve: `gatsby-source-opmlnote`,
      options: {
        url: `http://drummer.scripting.com/ScottHansonDE/papascott-de.opml`,
        name: 'posts',
        timezone: 'Europe/Berlin',
      },
    },

Is your plug-in running on Gatsby, parsing the opml and converting it to a format Gatsby understands (presumably markdown?)

That's correct!

@kevinctofel Saving the outline to GitHub from Drummer would be a logical next step, then all my content would be in the repository.

[kevinctofel]

@kevinctofel Saving the outline to GitHub from Drummer would be a logical next step, then all my content would be in the repository.

@papascott I suspect @scripting already has some of the plumbing done for that since he archives his blog posts to GitHub today IIRC. 😉

[scripting]

@kevinctofel -- i have lots of stuff that writes to github locations, i think it's an incredible resource and i want to use it more. but it's always been tricky from an authorization standpoint, it's easy for me to write code that posts stuff on my behalf, but for scripts, like the one @papascott is working on, i want them to use their own github resources.

an example app

See this folder on GH...

https://github.com/scripting/instantOutlines/tree/main/outlines/twitter

Those are a bunch of twitter accounts that an app of mine checks every minute, and posts whatever they post to a large opml file and upload it to that location.

spamming is a problem already

I could do that for every drummer user's blog.opml file, technically -- but there's another kind of security involved, not opening up all drummer users to spamming. it's amazing how quickly spammers show up these days. we had a long grace period in the early blogosphere, but this isn't like that anymore.

in the meantime if there are any apps people want to try out that require publicly accessible opml files that update regularly, this is a place to start. There's actually a lot of infrastructure built up around this idea.

[kevinctofel]

Right, you don’t want to be the GitHub gatekeeper and people should be able to connect Drummer to their own GitHub repos, if I’m understanding correctly. I believe that when I hooked up Netlify and GitHub together, they used this OAuth implementation which was a once and done from a user perspective. https://docs.github.com/en/developers/apps/building-oauth-apps/authorizing-oauth-apps#web-application-flow

If it were implemented in Drummer (in theory) people would connect their GitHub repo to Drummer from Drummer, at least as far as I can tell although as a coding n00b, I could very well be wrong. Drummer would then have an access token to write to the specific user’s GitHub repo on behalf of the user. And the user could revoke that access at any time, if desired, directly through GitHub. See image for the apps/services, I’ve authorized, for example.

Once connected, (again in theory) Drummer could save new blog posts to that GitHub repo, which Netlify would pick up as pull requests and kick off a build/publish process automatically.

Note that I’m not asking you to do this: I’m just providing some information and thinking out loud. ;)

[scripting]

@kevinctofel -- yes i think that's right. kind of like what we do with twitter, only with github. and i know from experience that's going to take a week from beginning to end. it has that feel.

in the meantime @papascott made it work without all that!!! 😀

[scotthansonde]

@kevinctofel If all you need to do is commit a single outline to GitHub on demand, we don't need any GitHub integration in Drummer. It could be done with a GitHub Action, and then a Drummer script that triggers the action over the GitHub API.

I've never used GitHub Actions before, but the following on a test repository seemed to work.

name: fetch-opml
on: [workflow_dispatch]
jobs:
  fetch-opml:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - shell: bash
        run: |
          git config --global user.name "GitHub Action"
          git config --global user.email "[email protected]"
          curl http://drummer.scripting.com/ScottHansonDE/papascott-de.opml -o papascott-de.opml
          git add papascott-de.opml
          git commit -m "Fetched at `date`" --allow-empty
          git push origin main

[kevinctofel]

Great thought and implementation @papascott! One question: Is the script specific to one particular GitHub repo? I don’t see any configuration on which repo so I’m assuming yes. I haven’t used GitHub Actions prior either, so I’m guessing the action is setup within a specific repo, which in this case, is the one you want to copy your OPML file to. If so, that handles the “user specific” GitHub repo questions yesterday. :)

[scripting]

@papascott -- I really want to understand how this works, but I don't.

esp if it's a way to get an opml file from Drummer-land into Github-land.

Is that what it's doing??

[scotthansonde]

I literally just learned this today, so I guess that makes me an expert! 😛 And yes, it does get an OPML file from Drummer-land to Github-land.

As I understand them, Github Actions are like scripts that are attached to a repository. They can be triggered, for example, by things that happen to the repository (like pushes or pull requests) or by external events (like an API call or hitting a button in the GitHub website).

The action I wrote starts up a virtual machine, checks out the repository, then runs some shell commands to fetch a public OPML file, save it in the repository, commit it to git, and push the commit to GitHub.

I would implement this for myself by writing a script that hits the GitHub API to trigger the action. But it's my first day doing this, I haven't gotten that far yet!

[scripting]

@papascott -- this is amazing. my head is exploding.

the part that really got me is "starts up a virtual machine" -- what can that virtual machine do? can it by any chance run a small Node app? if so, we are going to rock this so hard.

[scotthansonde]

@scripting Yes, even their "hello world" example installs node then installs an npm package. I've seen other examples that install node, run npm install, then runs a script from the package.json.

[scripting]

OMG -- https://www.youtube.com/watch?v=5kiNJcDG4E0

[alexjj]

This whole thread is awesome 🕶 Semi-related I found myself reading English's how to and when I read this I replaced LO with Drummer in my head :)

I've used github actions from other people's repos to build websites from content (you can make a template repo that others can use e.g) but never written any myself from scratch.

Just thinking out loud, but one could use drummer to write outlines, commit to github, a github action runs drummercms to generate the website, and commit it to a /public folder, which then uses Github pages to host with easy domain and https setup. The template index.html, code.js and style.css can all be kept in the github repo as well.

If someone wanted to integrate another outlining tool, e.g. a mobile app like Zavala, then they just need to work out of the github repo (maybe with working copy) and it's all together.

That's a lot of steps there and likely reworking of things but as a possible end state, it's an example. I want to play about with it all, just whenever I get round to making the time for it. Luckily drummer is pretty cool right now 😁

[SparkyT]

@kevinctofel If all you need to do is commit a single outline to GitHub on demand, we don't need any GitHub integration in Drummer. It could be done with a GitHub Action, and then a Drummer script that triggers the action over the GitHub API.

I've never used GitHub Actions before, but the following on a test repository seemed to work.

@papascott can you explain a bit more about how you set up the workflow_dispatch part? Are you getting a personal token from GitHub and incorporating that in the call to the GitHubAction?

I am also very excited about what GitHub Actions can do. Being able to schedule them as well as trigger them with API calls is great - my action doesn't yet use a workflow_dispatch, instead just running every 15 minutes (or on a manual push), downloading the files and only committing to the repo if the content has changed as opposed to looking at the timestamp.
I'm thinking of an action that can be triggered by Drummer as you describe and then scheduled as well just in case something happens.

# This is a basic workflow to help you get started with Actions
# downloads http://drummer.scripting.com/davewiner/drummer/changeNotes.opml and other files every 15 minutes then commits any content changes to the repo 

name: ScriptingDrummerBackups

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch plus on a schedule as set below
  push:
    branches: [ main ]
  schedule:
    - cron:  '*/15 * * * *'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check out repo
      uses: actions/checkout@v2
      with:
        fetch-depth: 0
    - name: Download Feed
      run: |-
        #  Getting our files, really I should move these into a separate list
        curl "http://drummer.scripting.com/davewiner/drummer/changeNotes.opml" -o opml/changeNotes.opml
        curl http://scripting.com/rss.xml -o rss/scripting.com-rss.xml
        curl "https://raw.githubusercontent.com/scripting/instantOutlines/main/outlines/twitter/davewiner.opml" -o opml/davewiner.opml
    - name: Commit and push changes
      run: |-
        #
        #  based on https://thechels.uk/saving-json-github-action
        #  git diff at the start to clear comparisons based on timestamps or something as the files are downloaded
        #   each time but may not have changed in content
        #  [!-z.... line checks for changes and only runs the git add and git commit if there sre some 
        #
        git diff
        git config user.name "Automated"
        git config user.email "[email protected]"
        [ ! -z "$(git status --porcelain=v1 2>/dev/null)" ] && (git add -A && git commit -m "Updated with latest")        
        git push

[scotthansonde]

@SparkyT Yeah, something like that. You can only set a token for a user that is not restricted to a single repository, so I created a machine user that collaborates on the repository. I use a token for the machine user to start the action. I've been using the new http.client verb in Drummer, something like this:

var actionURL = "https://api.github.com/repos/papascott/test-github-actions/actions/workflows/learn-github-actions.yml/dispatches"
var theRequest = {
	type: "POST",
	url: actionURL,
	data: '{"ref": "main"}',
	headers: {"Authorization": "Bearer "+ githubToken} 
	};
console.log (http.client (theRequest, true))

Storing the token in Web Drummer is probably not yet secure, but should be OK in Electric Drummer.