Add custom loss functions and a R/W state matrix #936
Conversation
Add support for loss functions in CPU-based neural networks
* IT WORKS FOR GPU NOW, DON'T TOUCH ANYTHING YET LMAO * Bugs noted: It throws if you don't provide a loss function; should be an extremely trivial fix, but I had this working, broke it, panicked, and had to fix it over the course of half an hour. So please, understand that I HAD TO COMMIT!!!
`NeuralNetworkGPU` no longer throws an error when a loss function is not provided
Allow for using the same loss function between both CPU and GPU neural networks
This reverts commit a762a48.
Add `updateRAM` as: * property `NeuralNetwork.updateRAM: RAMFunction` * option `NeuralNetworkOptions.updateRAM` * option `NeuralNetworkTrainOptions.updateRAM` *(temporary sets `this.updateRAM` for the duration of the training session)*
|
Working on the failed CI lint tasks rn while my food is in the microwave!! I'm sorry, I'm just so excited for brain.js to have this feature!!! I've been wishing we could do this in brain for literal years now 🥰 💕 |
|
All is well except for that one CI / Build error for Mac 💕 💕 💕 😊 💕 💕 💕 |
Update tests and documentation to reflect recent change from `Autoencoder.includesAnomalies` to `Autoencoder.likelyIncludesAnomalies` as the method makes no API guarantee to detect anomalies with 100% accuracy
|
Many apologies for taking a brief hiatus! A lot has happened in my personal life. I'm starting work on implementing these features within the |
|
Could you clean up conflicts, and we can get this in asap? Thank you for your patience. |
Absolutely!! I'm so sorry that I've taken so long; my health hasn't been the best. I'm supposed to go in for surgery tomorrow, so I'll see if I can rush it tonight 😊 If not, I'll make sure to resolve them as soon as I'm in working condition again, which shouldn't take more than a few days |
|
My health is permitting near-full-time work again. I'll update you once we're ready for merging 😊 |
Preface
I'm still documenting all of the changes, so please don't bonk me with the dumb-dumb-document stick! I promise to be a good girl and finish documenting all of the changes I made very shortly. I'll also make sure to add unit tests for the loss functions and R/W state matrices; I just need a moment to rest up and eat a meal first 💞
Description
I added support for defining custom loss functions, as well as a R/W state matrix to serve as RAM to allow for training by ruleset in addition to, or even in replacement of, a pre-existing set of training data. In my own benchmarks, a custom loss function designed to train for XOR improved training on the XOR data set by approximately 100% for
NeuralNetwork(CPU) and 400% forNeuralNetworkGPU(GPU). This is, of course, a really specific example. However, it's just that; an example. This paves the way for any custom loss function to be defined. On GPU, you obviously are limited to values you pass through theNeuralNetwork.ramproperty, which is the R/W state matrix I previously mentioned. However, on CPU, you're honestly not limited by anything, as the function is called from the CPU, not the GPU, meaning you could hypothetically even go so far as to involve API calls in the loss function calculation if you really wanted to so long as your internet bandwidth permits without becoming too major of a bottleneck in training times.Motivation and Context
discussion
issue
How Has This Been Tested?
I ran the test suite. There's currently two failed tests that are related to these changes; the other 3-or-so failed tests are the same RNN tests that failed on a fresh copy of the master branch.
Of the 2 failed tests, the first one is related to the
AEclass; these changes actually break the currentAEclass due to how it was implemented. This is a non-issue, as custom loss functions give us a far better way to implement autoencoders.The second failed test, however, I'm unsure of how to deal with it as of the current moment (likely a side effect of writing code daily for over a month on end with little food, water, or sleep because poverty is yikers and kind of makes surviving not so easy lol). I'll probably figure it out after some well-needed rest and food (I'm making a healthy meal for dinner rn to replenish my body 💞)
Screenshots (if appropriate):
Hyperfine benchmark results
nn-cpu-loss-NeuralNetworkwith a custom loss functionnn-cpu-NeuralNetworkwithout a custom loss functionnn-loss-NeuralNetworkGPUwith a custom loss functionnn-NeuralNetworkGPUwithout a custom loss functionWhy is the CPU so much faster?
Because the model used in the benchmarks is super tiny (2 inputs, 3 neurons in a single hidden layer, 1 output). Check out this issue for more information if you're interested.
Types of changes
Breaking changes
The only breaking change is that the current autoencoder implementation will break. This is actually a neutral/good thing, as this feature adds a much better way to implement autoencoders anyways, making this singular breaking change a moot point seeing as we're still currently in a beta period before the next major release, and theAEclass was literally just added not even a week or two ago. I'll be working on porting the new autoencoder implementation I wrote back into brain.js from the li'l sandbox I made for devtesting brain.js/gpu.js/related and hopefully it should be finished before pull time.AE) has been split into two classes:AutoencoderandAutoencoderGPU.AutoencoderextendsNeuralNetworkand runs exclusively on the CPU.AutoencoderGPUextendsNeuralNetworkGPUand runs on the GPU if one is present, and otherwise falls back to the CPU.Author's Checklist:
Reviewer's Checklist: