extensions.ipynb


        {
            "cells": [
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": null, "outputs": [],
           "source": ["// Google Colab only: uncomment and run the following to install dotnet and the F# kernel\n",
"// !bash \u003c(curl -Ls https://raw.githubusercontent.com/gbaydin/scripts/main/colab_dotnet6.sh)\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": null, "outputs": [],
           "source": ["// Import DiffSharp package\n",
"#r \"nuget: DiffSharp-lite,1.0.7\"\n",
"\n",
"// Set dotnet interactive formatter to plaintext\n",
"Formatter.SetPreferredMimeTypesFor(typeof\u003cobj\u003e, \"text/plain\")\n",
"Formatter.Register(fun (x:obj) (writer: TextWriter) -\u003e fprintfn writer \"%120A\" x )\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["[![Binder](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DiffSharp/diffsharp.github.io/blob/master/extensions.ipynb)\u0026emsp;\n",
"[![Binder](img/badge-binder.svg)](https://mybinder.org/v2/gh/diffsharp/diffsharp.github.io/master?filepath=extensions.ipynb)\u0026emsp;\n",
"[![Script](img/badge-script.svg)](extensions.fsx)\u0026emsp;\n",
"[![Script](img/badge-notebook.svg)](extensions.ipynb)\n",
"\n",
"# Extending DiffSharp\n",
"\n",
"DiffSharp provides most of the essential operations found in tensor libraries such as [NumPy](https://numpy.org/), [PyTorch](https://pytorch.org/), and [TensorFlow](https://www.tensorflow.org/). All differentiable operations support the forward, reverse, and nested differentiation modes.\n",
"\n",
"When implementing new operations, you should prefer to implement these as compositions of existing DiffSharp [Tensor](https://diffsharp.github.io/reference/diffsharp-tensor.html) operations, which would give you differentiability out of the box.\n",
"\n",
"In the rare cases where you need to extend DiffSharp with a completely new differentiable operation that cannot be implemented as a composition of existing operations, you can use the provided extension API.\n",
"\n",
"## Simple elementwise functions\n",
"\n",
"If the function you would like to implement is a simple elementwise function, you can use the [UnaryOpElementwise](https://diffsharp.github.io/reference/diffsharp-unaryopelementwise.html) or [BinaryOpElementwise](https://diffsharp.github.io/reference/diffsharp-binaryopelementwise.html) types to define your function and its derivatives. The forward, reverse, and nested differentiation rules for the function are automatically generated by the type. The documentation of these two types detail how they should be instantiated.\n",
"\n",
"Let\u0027s see several examples.\n",
"\n",
"$f(a) = \\mathrm{sin}(a)$, with derivative $\\frac{\\partial f(a)}{\\partial a} = \\mathrm{cos}(a) \\;$.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 2, "outputs": [],
           "source": ["open DiffSharp\n",
"\n",
"type Tensor with\n",
"    member a.sin() = \n",
"        Tensor.Op\n",
"            { new UnaryOpElementwise(\"sin\") with \n",
"                member _.fRaw(a) = a.SinT()\n",
"                member _.dfda(a,f) = a.cos()\n",
"            }\n",
"            (a)\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["$f(a) = \\mathrm{log}(a)$, with derivative $\\frac{\\partial f(a)}{\\partial a} = 1/a \\;$.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 3, "outputs": [],
           "source": ["type Tensor with\n",
"    member a.log() =\n",
"        Tensor.Op\n",
"            { new UnaryOpElementwise(\"log\") with\n",
"                member _.fRaw(a) = a.LogT()\n",
"                member _.dfda(a,f) = 1/a\n",
"            }\n",
"            (a)\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["$f(a, b) = ab$, with derivatives $\\frac{\\partial f(a, b)}{\\partial a} = b$, $\\frac{\\partial f(a, b)}{\\partial b} = a \\;$.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 4, "outputs": [],
           "source": ["type Tensor with\n",
"    member a.mul(b) =\n",
"        Tensor.Op\n",
"            { new BinaryOpElementwise(\"mul\") with\n",
"                member _.fRaw(a,b) = a.MulTT(b)\n",
"                member _.dfda(a,b,f) = b\n",
"                member _.dfdb(a,b,f) = a\n",
"            }\n",
"            (a,b)\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["$f(a, b) = a^b$, with derivatives $\\frac{\\partial f(a, b)}{\\partial a} = b a^{b-1}$, $\\frac{\\partial f(a, b)}{\\partial b} = a^b \\mathrm{log}(a) \\;$. Note the use of the argument `f` in the derivative definitions that makes use of the pre-computed value of $f(a, b) = a^b$ that is available to the derivative implementation.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 5, "outputs": [],
           "source": ["type Tensor with\n",
"    member a.pow(b) =\n",
"        Tensor.Op\n",
"            { new BinaryOpElementwise(\"pow\") with\n",
"                member _.fRaw(a,b) = a.PowTT(b)\n",
"                member _.dfda(a,b,f) = b * f / a  // equivalent to b * a.pow(b-1)\n",
"                member _.dfdb(a,b,f) = f * a.log()  // equivalent to a.pow(b) * a.log()\n",
"            }\n",
"            (a,b)\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["## General functions\n",
"\n",
"For more complicated functions, you can use the most general way of defining functions using the [UnaryOp](https://diffsharp.github.io/reference/diffsharp-unaryop.html) or [BinaryOp](https://diffsharp.github.io/reference/diffsharp-binaryop.html) types, which allow you to define the full forward and reverse mode differentiation rules. The documentation of these two types detail how they should be instantiated.\n",
"\n",
"Let\u0027s see several examples.\n",
"\n",
"$f(A) = A^{\\intercal}$, with the forward derivative propagation rule $\\frac{\\partial f(A)}{\\partial X} = \\frac{\\partial A}{\\partial X} \\frac{\\partial f(A)}{\\partial A} = (\\frac{\\partial A}{\\partial X})^{\\intercal}$ and the reverse derivative propagation rule $\\frac{\\partial Y}{\\partial A} = \\frac{\\partial Y}{\\partial f(A)} \\frac{\\partial f(A)}{\\partial A} = (\\frac{\\partial Y}{\\partial f(A)})^{\\intercal} \\;$.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 6, "outputs": [],
           "source": ["type Tensor with\n",
"    member a.transpose() =\n",
"        Tensor.Op\n",
"            { new UnaryOp(\"transpose\") with\n",
"                member _.fRaw(a) = a.TransposeT2()\n",
"                member _.ad_dfda(a,ad,f) = ad.transpose()\n",
"                member _.fd_dfda(a,f,fd) = fd.transpose()\n",
"            }\n",
"            (a)\n"]
          }
,
          {
           "cell_type": "markdown",
           "metadata": {},
           
           "source": ["$f(A, B) = AB$, with the forward derivative propagation rule $\\frac{\\partial(A, B)}{\\partial X} = \\frac{\\partial A}{\\partial X} \\frac{\\partial f(A, B)}{\\partial A} + \\frac{\\partial B}{\\partial X} \\frac{\\partial f(A, B)}{\\partial B} = \\frac{\\partial A}{\\partial X} B + A \\frac{\\partial B}{\\partial X}$ and the reverse propagation rule $\\frac{\\partial Y}{\\partial A} = \\frac{\\partial Y}{\\partial f(A, B)} \\frac{\\partial f(A, B)}{\\partial A} = \\frac{\\partial Y}{\\partial f(A, B)} B^{\\intercal}$, $\\frac{\\partial Y}{\\partial B} = \\frac{\\partial Y}{\\partial f(A, B)} \\frac{\\partial f(A, B)}{B} = A^{\\intercal} \\frac{\\partial Y}{\\partial f(A, B)} \\;$.\n",
"\n"]
          }
,
          {
           "cell_type": "code",
           "metadata": {},
            "execution_count": 7, "outputs": [],
           "source": ["type Tensor with\n",
"    member a.matmul(b) =\n",
"        Tensor.Op\n",
"            { new BinaryOp(\"matmul\") with\n",
"                member _.fRaw(a,b) = a.MatMulTT(b)\n",
"                member _.ad_dfda(a,ad,b,f) = ad.matmul(b)\n",
"                member _.bd_dfdb(a,b,bd,f) = a.matmul(bd)\n",
"                member _.fd_dfda(a,b,f,fd) = fd.matmul(b.transpose())\n",
"                member _.fd_dfdb(a,b,f,fd) = a.transpose().matmul(fd)\n",
"            }\n",
"            (a,b)\n"]
          }],
            "metadata": {
            "kernelspec": {"display_name": ".NET (F#)", "language": "F#", "name": ".net-fsharp"},
            "langauge_info": {
        "file_extension": ".fs",
        "mimetype": "text/x-fsharp",
        "name": "C#",
        "pygments_lexer": "fsharp",
        "version": "4.5"
        }
        },
            "nbformat": 4,
            "nbformat_minor": 1
        }
        