This repo contains a very basic implementation of the model Funcformer as described in the blogpost 'Generalized Transformers from Applicative Functors'.
Two examples are given in poisson_example1.mp4 and poisson_example2.mp4.
The script solve_poisson.py gives an example of solving the Poisson equation using Funcformer for random smooth functions with fixed boundary conditions.
Running it will produce some output files that can be viewed using the plots.ipynb notebook.
The chebformer.py file includes the PyTorch module itself so you can use it for other tasks. The interface is designed to be similar to other PyTorch models, but see solve_poisson.py for an example on how to instantiate it.
Dependencies:
torchtorch_dct
This implementation has some problems. Mostly, it fails to converge quite a lot of the time. If this happens, you should try:
- Decreasing the learning rate drastically (i.e 1e-5 is not unreasonable)
- Decreasing the batch size (I've found it works best with batch sizes 4-16)
- Using the SGD optimizer instead of Adam or other optimizers. I don't know why this helps, but it does!