00_core.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "#default_exp core\n",
    "#all_slow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Creating tweetrel\n",
    "> and an overview of its API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from fastcore.utils import *\n",
    "from fastcore.script import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to create this library, we will start out by following the initial steps of the [nbdev tutorial](https://nbdev.fast.ai/tutorial.html) to create a basic project structure. We also need to install [ghapi](https://ghapi.fast.ai/), by following the instructions on that link.\n",
    "\n",
    "Once we have a repo cloned, based on the `nbdev_template` template, we run in our terminal:\n",
    "\n",
    "```bash\n",
    "gh-create-workflow tweet release --contexts secrets\n",
    "```\n",
    "\n",
    "Note that we add `--contexts secrets` because we'll need access to the [secrets context](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow), which we'll be using to store our Twitter API keys.\n",
    "\n",
    "The basic workflow skeleton that this creates is:\n",
    "\n",
    "```bash\n",
    ".github/workflows/tweet-release.yml\n",
    ".github/scripts/build-tweet-release.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Twitter authentication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to send tweets, we'll send to use the Twitter API -- we'll use the `tweepy` library for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "import tweepy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll need a [Twitter API login](https://developer.twitter.com/en/apply-for-access). If you need to send Tweets on behalf of a different user to the one attached to your login, you'll need to authenticate with Twitter as that user. Run `tweetrel-auth` in your terminal and follow the instructions to get those access tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "@call_parse\n",
    "def auth():\n",
    "    \"Function for `tweetrel-auth` CLI command\"\n",
    "    cons = input('Enter consumer key and secret, separated by a space: ').split()\n",
    "    auth = tweepy.OAuthHandler(*cons)\n",
    "    print('Go to this URL and login:')\n",
    "    print(auth.get_authorization_url())\n",
    "    code = input('Paste code here: ')\n",
    "    print('Your keys are:')\n",
    "    print(' '.join(auth.get_access_token(code)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Twitter part of this tutorial isn't the main thing we want to explain, but the thing to note carefully here is how to access *secrets* in your workflow.\n",
    "\n",
    "The Twitter API requires authentication, so we'll store our details in a GitHub secret. We'll just use one secret, we our keys stored space delimited. We generally use an [organization secret](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-an-organization), so that we can update secrets in one place as needed.\n",
    "\n",
    "Contexts are passed to ghapi as json-encoded environment variables. Each variable name starts with `CONTEXT_` -- so for instance the `secrets` context is `CONTEXT_SECRETS`. In order to simulate this when testing, ensure that you have set an appropriate environment variable before importing `ghapi`. We'll put our JSON encoded secrets for testing into a `.secrets` file (which we add to `.gitignore` so it won't be pushed to our repo). We put the four parts (`consumer_key`, `consumer_secret`, `access_token`, and `access_token_secret`) into a single space delimited secret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['CONTEXT_SECRETS'] = Path('.secrets').read_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can import `ghapi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "from ghapi.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you now check the `context_secrets` variable, you should find your secrets available as an `AttrDict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert 'TWITTER' in context_secrets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll create a function to unpack our keys, tokens, and secrets for Twitter auth, and login to the API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "def twitter_api():\n",
    "    consumer_key,consumer_secret,access_token,access_token_secret = context_secrets.TWITTER.split()\n",
    "    auth = tweepy.OAuthHandler(consumer_key,consumer_secret)\n",
    "    auth.set_access_token(access_token,access_token_secret)\n",
    "    return tweepy.API(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test our login by creating and deleting a tweet:\n",
    "\n",
    "```python\n",
    "import time\n",
    "twapi = twitter_api()\n",
    "stat = twapi.update_status(\"Please ignore - testing API\")\n",
    "time.sleep(1)\n",
    "twapi.destroy_status(stat.id)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Responding to the `release` event"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a sample `release` event payload as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "example_rel = example_payload(Event.release)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['action', 'release', 'repository', 'sender'], 'published')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(example_rel), example_rel.action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `release` section has the following keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'url, assets_url, upload_url, html_url, id, node_id, tag_name, target_commitish, name, draft, author, prerelease, created_at, published_at, assets, tarball_url, zipball_url, body'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "', '.join(example_rel.release)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a function that formats a tweet based on this information, which will take the template from an environment variable if available, otherwise will use a reasonable default:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "def tweet_text(payload):\n",
    "    if 'workflow' in payload: payload = example_rel\n",
    "    def_tmpl = \"New #{repo} release: v{tag_name}. {html_url}\\n\\n{body}\"\n",
    "    tweet_tmpl = os.getenv('TWEETREL_TEMPLATE', def_tmpl)\n",
    "    rel = payload.release\n",
    "    owner,repo = payload.repository.full_name.split('/')\n",
    "    res = tweet_tmpl.format(repo=repo, tag_name=rel.tag_name, html_url=rel.html_url, body=rel.body)\n",
    "    return res if len(res)<=280 else (res[:279] + \"…\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New #Hello-World release: v0.0.1. https://github.com/Codertocat/Hello-World/releases/tag/0.0.1\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(tweet_text(example_rel))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample release payload from GitHub happens to have an empty `body`, but other than that, this looks good.\n",
    "\n",
    "That's all we need to create our function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "def send_tweet():\n",
    "    payload = context_github.event\n",
    "    if 'workflow' in payload: payload = example_payload(Event.release)\n",
    "    if payload.action == 'published': return twitter_api().update_status(tweet_text(payload))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pop this into our python script, along with `send_tweet()`. Since we're using nbdev, we can enter the following in our terminal to do all that:\n",
    "\n",
    "```bash\n",
    "nbdev_build_lib\n",
    "cp tweetrel/core.py .github/scripts/build-tweet-release.py\n",
    "echo -e \"\\nsend_tweet()\" >>  .github/scripts/build-tweet-release.py\n",
    "```\n",
    "\n",
    "After we push to GitHub, we can test it out in the same way we showed in the [GitHub Actions tutorial](https://ghapi.fast.ai/tutorial_actions.html), first by logging in and getting a reference to our workflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api = GhApi(owner='fastai', repo='tweetrel', token=github_token())\n",
    "wf = api.actions.get_workflow('tweet-release.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and then running it:\n",
    "\n",
    "```python\n",
    "api.actions.create_workflow_dispatch(wf.id, ref='master')\n",
    "```\n",
    "\n",
    "After you run this, you should find the workflow appears in your \"Actions\" tab on GitHub, and a tweet will appear in your twitter timeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some little improvements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One issue is that currently you'll see three workflows being triggered on the Actions tab in GitHub. That's because the \"created\" and \"released\" types are resulting in a trigger, as well as \"published\". Our function is checking which is being used, and only tweeting for \"published\", but the runs are still being recorded. This isn't necessarily a big problem, but if you'd like things to be cleaner, you can edit the `tweet-release.yml` file to add the following line after the `release:` line:\n",
    "\n",
    "```bash\n",
    "types: [published]\n",
    "```\n",
    "\n",
    "It's also possible to do a more complete end-to-end test by actually making a release, checking that a new run is created, and checking the result of that run:\n",
    "\n",
    "```python\n",
    "n_runs = api.actions.list_workflow_runs(wf.id).total_count\n",
    "try:\n",
    "    rel = api.create_release('test', body='body')\n",
    "    time.sleep(30)\n",
    "    runs = api.actions.list_workflow_runs(wf.id)\n",
    "    test(runs.total_count, n_runs, gt)\n",
    "    test_eq(runs.workflow_runs[0].conclusion, 'success')\n",
    "finally: api.delete_release(rel)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distributing your new Action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above steps are all that's needed to create a Python-based workflow for use in a single project. To make it accessible for any project (including other people's projects), we need to distribute it. The easiest way to do that is with `pip`. `pip` can install modules directly from GitHub, or from the PyPi repository.\n",
    "\n",
    "Our first step is to add a command that makes it easy to install the `tweetrel` workflow into a project. We can use `fastcore.script.call_parse` to create a function that can be run from the terminal.\n",
    "\n",
    "In the function, we'll use `fill_workflow_templates` instead of `gh-create-workflow` because it allows us to customize each part of the YAML workflow file exactly as we need it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#exports\n",
    "@call_parse\n",
    "def install():\n",
    "    fill_workflow_templates(\n",
    "        name='tweet', event=\"release:\\n  types: [published]\",\n",
    "        run='pip install -Uq tweetrel',\n",
    "        context=env_contexts('secrets'),\n",
    "        script=\"import tweetrel\\ntweetrel.send_tweet()\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we just used `tweetrel` in the \"pip install\" line, which assumes that the project has been uploaded to pypi (which you can do by typing `make release` in your terminal, if you use nbdev or [pypi_template](https://github.com/fastai/pypi_template). If you are just installing from GitHub directly, you'd use this instead:\n",
    "\n",
    "```python\n",
    "run='pip install -Uq git+https://github.com/fastai/tweetrel.git',\n",
    "```\n",
    "\n",
    "Since `install` uses `call_parse`, we can run it as a standard Python function, so we can test it by executing the function:\n",
    "\n",
    "```python\n",
    "install()\n",
    "```\n",
    "\n",
    "Alternatively, we can run it through the CLI, by first adding to our `settings.ini`:\n",
    "\n",
    "```bash\n",
    "console_scripts = tweetrel-install=tweetrel.core:install\n",
    "```\n",
    "\n",
    "...and then installing our library locally:\n",
    "\n",
    "```bash\n",
    "pip install -e .\n",
    "```\n",
    "\n",
    "...and finaly running the command:\n",
    "\n",
    "```bash\n",
    "tweetrel-install\n",
    "```\n",
    "\n",
    "If this all works, then you can run `make release` to submit your action to pypi and conda, making it available for anyone to use in their workflows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By following these steps, we have created a Python-based GitHub Action, which can be installed using pip or conda, and can be added to any repo with a single command. See the overview page for this module to see how we document this for end users of the action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 00_core.ipynb.\n",
      "Converted index.ipynb.\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "from nbdev.export import notebook2script\n",
    "notebook2script()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}