Dear Reader, In this post I would like to summarize the matrix multiplication algorithms I am using in my neural network library – hopefully they will come handy for some of you. The data types Every SIMD instruction works on a vector of data in parallel. This vector is a row in our matrix. __m128: […]
I woke up at midnight and thought – what if I tried to further optimise the SSE block multiplier? I started browsing the intrinsics documentation from Intel, and I quickly came to the conclusion that it would not be too hard to implement AVX intrinsics instead of SSE, which allows blocks of 8×8 single precision […]
Dear Reader, I have been through a very productive weekend, I have managed to implement a simple matrix partitioning algorithm, and also a matrix multiplier function that multiplies 4×4 matrices using SSE intrinsics. Partitioning is not yet as efficient as I know it could be, because it creates 4×4 blocks no matter what. Even in […]
Dear Reader I have been working on a multithreaded implementation of the Backpropagation algorithm. The most computationally intensive parts of a learning iteration are the forward-propagation and the backpropagation steps which are used in all algorithms to determine the gradients. The main difference between these algorithms is how these gradients are used to update the […]
Ok, my apologies in advance if you expected artificial intelligence agents playing with each other in this post, I couldn’t help myself when I wrote the title. This post is about comparing the speeds of two significantly different implementations of the same neural network algorithm. I’ve just done a very simple performance comparison between my […]
In this short entry I would like to talk about the details behind the “magical” buzzwords spreading these days on the Machine Learning and Big Data scenes. I am sure the marketing and sales department at NVidia is making great efforts to put expressions like “tensor cores” to be the topic of common talk – […]
Dear Reader, This project is currently in progress but I thought I would publish it anyway. I have created a modern C++ DLL from the code extracted from the Borland C++ Builder project, as my 15 year old code is hardly going to be useful for anyone these days. https://github.com/bulyaki/NNTrainerLib What this library currently does […]
Dear Reader, I decided to publish an old version of the RProp algorithm I wrote back in 2003 that demonstrates how the algorithm modified to use matrix algebra works. The demo can be downloaded from here, both source files and binaries: https://github.com/bulyaki/NeuralNetworkTrainer Originally this version was lacking a couple of libraries and I could not […]
Since the RProp algorithm uses if/else conditional statements while determining update values some special helper matrix-functions and helper matrices must be introduced. These functions will allow us to express the conditional statements more elegantly, only using matrix operations. Some matrices needed to make the above functions work: D: decision matrix, M: meta-gradient matrix, U: update […]
In a multi-layered neural network weights and neural connections can be treated as matrices, the neurons of one layer can form the columns, and the neurons of the other layer can form the rows of the matrix. The figure below shows a network and its parameter matrices. The meanings of vectors and matrices above: ninl […]