I decided to publish an old version of the RProp algorithm I wrote back in 2003 that demonstrates how the algorithm modified to use matrix algebra works.
The demo can be downloaded from here, both source files and binaries:
Right now I don’t have any means to make changes to it as I haven’t used Borland C++ builder for ages.
Here is how to use it:
First you have to click on the “Load CSV” button. For this purpose you need a CSV file that contains both the inputs and the desired targets (training patterns) for the Neural Network.
Each column is either an input vector or an output vector, inputs come first, outputs afterwards. So they should look something like this (this is a snippet from the Powers.csv file contained in the repository):
The piece of data above has one inputs, and 5 outputs. The inputs are numbers from -75.0 to -74.0 with 0.1 increments. The outputs are second, third, …, sixth powers of the inputs.
Out1 = In^2
Out2 = In^3
Out3 = In^4
Out4 = In^5
Out5 = In^6
Then you will have to specify how many inputs and outputs you wish to connect to your Neural Network:
Now that we know the number of inputs and outputs, you have to specify the dimensions of your neural network. Let’s create a network with two hidden layers, with each layer containing 10 neurons:
We can leave the number of iterations 1000. Then click on the “Start” button:
The graph in the middle shows the regression of the output and the training vectors. In the powers.csv file the first output is a parabola function (y=x^2, x=[-75.0 … +75.0]).
On the above photo you can see how both sides of the parabola is taking shape, and eventually it will reach a close to 45 degree straight line (well, more or less).
Obviously neural networks should not be used to for functions where we know the connection between input and output variables, but such functions are very useful for testing our network and our algorithm.
Please note that I created the above application about 14 years ago, so the code in the repository is NOT a faithful representation of my programming and design abilities. Also, if you find bugs in the code you are welcome to fix them yourself, but I will not be able to make any further modifications to it. This application had lots of versions, some of which were able to display the matrices of the network in 3D using OpenGL, some allowing you to create various network topologies using a bespoke designer UI, but all these versions were unfinished and probably buggy. The one in the repository is probably the cleanest source I have that solely concentrates on the matrix form of the RProp algorithm itself.
I am already working on a C# version of this algorithm which I’m planning to publish in the coming weeks. Also once ready, I plan to republish these blog posts as one long article on CodeProject.