关于Differential Programming/Differentiable Programming/DDP/Automatic differentiation/Differentiable neural computer的论文，文章，教程，幻灯片和项目的列表。
-
What is differential programming? How is it related to functional programming?
Differential programming, also known as Dynamic Differential Programming (DDP) is an optimization procedure for path planning used in control theory and robotics:
DDP is an algorithm that solves locally-optimal trajectories given a cost function over some space. In essence it works by locally-approximating the cost function at each point in the trajectory. It uses this approximation to finds the optimal change to the trajectory (via a set of actions) that minimizes some cost metric (e.g. cumulative cost). In the limit it converges to the optimal trajectory. 摘自https://www.quora.com/What-is-differential-programming-How-is-it-related-to-functional-programming -
'Deep Learning est mort. Vive Differentiable Programming'
Yann LeCun:OK, Deep Learning has outlived its usefulness as a buzz-phrase.
Deep Learning est mort. Vive Differentiable Programming!
Yeah, Differentiable Programming is little more than a rebranding of the modern collection Deep Learning techniques, the same way Deep Learning was a rebranding of the modern incarnations of neural nets with more than two layers.
But the important point is that people are now building a new kind of software by assembling networks of parameterized functional blocks and by training them from examples using some form of gradient-based optimization.
An increasingly large number of people are defining the networks procedurally in a data-dependent way (with loops and conditionals), allowing them to change dynamically as a function of the input data fed to them. It's really very much like a regular progam, except it's parameterized, automatically differentiated, and trainable/optimizable. Dynamic networks have become increasingly popular (particularly for NLP), thanks to deep learning frameworks that can handle them such as PyTorch and Chainer (note: our old deep learning framework Lush could handle a particular kind of dynamic nets called Graph Transformer Networks, back in 1994. It was needed for text recognition).
People are now actively working on compilers for imperative differentiable programming languages. This is a very exciting avenue for the development of learning-based AI.
Important note: this won't be sufficient to take us to "true" AI. Other concepts will be needed for that, such as what I used to call predictive learning and now decided to call Imputative Learning. More on this later
“Differentiable programming”: This is the idea of viewing a program (or a circuit) as a graph of differentiable modules that can be trained with backprop. This points towards the possibility of not just learning to recognize patterns (as with feed-forward neural nets) but to produce algorithms (with loops, recursion, subroutines, etc). There are a few papers on this from DeepMind, FAIR, and others, but it’s rather preliminary at the moment. “差分编程”：这是将程序（或电路）视为可以用backprop训练的可微模块的图形的想法。 这表明不仅可以学习识别模式（如前馈神经网络），还可以生成算法（包括循环，递归，子程序等）。 有较少来自于DeepMind，FAIR和其他的文章，但目前处于初步阶段。
-
Differential Programming
[1]Atkeson C G, Stephens B J. Random sampling of states in dynamic programming[J]. IEEE Trans Syst Man Cybern B Cybern, 2008, 38(4):924-929.
[2]Fan D D, Theodorou E A. Differential Dynamic Programming for time-delayed systems[C]// Decision and Control. IEEE, 2016:573-579.
[3]Pan Y, Theodorou E A. Probabilistic differential dynamic programming[J]. Advances in Neural Information Processing Systems, 2014, 3:1907-1915.
[4]Yamaguchi A, Atkeson C G. Neural networks and differential dynamic programming for reinforcement learning problems[C]// IEEE International Conference on Robotics and Automation. IEEE, 2016.
[5]Mayne D Q. Differential Dynamic Programming - A Unified Approach to the Optimization of Dynamic Systems[J]. Control & Dynamic Systems, 1973, 10:179-254.
[6]DehazeGAN: When Image Dehazing Meets Differential Programming,IJCAI-18
更新于2019.6.19
[7][Zhang_ISTA-Net_Interpretable_Optimization-Inspired_CVPR_2018_paper]
[8][ISTA-Net Iterative Shrinkage-Thresholding Algorithm Inspired Deep Network for Image Compressive Sensing]
[9][Deep ADMM-Net for Compressive Sensing MRI]
[10][Deep Unfolding Model-Based Inspiration of Novel Deep Architectures]
[11][Differentiable Linearized ADMM]
[12]Differentiable Optimization-Based Modeling for Machine Learning -
Differentiable Programming
[1]Singh C. Optimality conditions in multiobjective differentiable programming[J]. Journal of Optimization Theory & Applications, 1988, 57(2):369-369.
[2]Wang F, Wu X, Essertel G, et al. Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator[J]. 2018.
[3]Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, Jonathan Ragan-Kelley. Differentiable Programming for Image Processing and Deep Learning in Halide. ACM Transactions on Graphics 37(4) (Proceedings of ACM SIGGRAPH 2018) code
[4]A Language and Compiler View on Differentiable Programming ICLR 2018 Workshop Track
[5]Gaunt A L, Brockschmidt M, Kushman N, et al. Differentiable Programs with Neural Libraries[J]. PMLR.2017.
[6]k-meansNet: When k-means Meets Differentiable Programming arxiv 2018
[7]Differentiable Learning of Logical Rules for Knowledge Base Reasoning NIPS 2017
[8]Programming With a Differentiable Forth Interpreter
[9]Efficient Differentiable Programming in a Functional Array-Processing Language -
Automatic differentiation [1]Baydin A G, Pearlmutter B A, Radul A A, et al. Automatic differentiation in machine learning: a survey[J]. Computer Science, 2015(February).
-
Differential Programming
[1]Atkeson_Stephens_Random Sampling of States in Dynamic Programming_NIPS 2007
[2]Stochastic Differential Dynamic Programming_Theodorou_ACC2010 -
Differentiable Programming
[1]Differentiable Programming
[2]Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, Jonathan Ragan-Kelley. Differentiable Programming for Image Processing and Deep Learning in Halide. ACM Transactions on Graphics 37(4) (Proceedings of ACM SIGGRAPH 2018)
[3][Deep learning frameworks and differentiable programming](Deep learning frameworks and differentiable programming)
[4]Differentiable Functional Programming
[5]Getting started with Differentiable Programing
[6]Deep Learning, differentiable programming, and software 2.0
-
Differential Programming
-
Differentiable Programming
[1]Differentiable Programming: A Semantics Perspective
[2]Tensorlang, a differentiable programming language based on TensorFlow
[3]Differentiable programming in Gluon and Python (For not only for medical image analysis)
[4]torchbearer
[5]Computers May Be Closer to Learning Common Sense Than We Think