Artificial neural networks (ANNs) – are conceptually simple; the combination of inputs and weights in a classical ANN can be represented as a single matrix product operation followed by an elementwise nonlinearity. However, as the number of learned parameters increases, it becomes very difficult to train these networks effectively. Most… Read More »Biologically-plausible learning rules for artificial neural networks
Reproduced left to right from [8,10,1] Within a 5 day span in October, 4 papers came out that take a significantly different approach to AI hierarchical networks. They are all inspired by biological principles to varying degrees. It’s exciting to see different ways of thinking. Particularly at a time… Read More »New approaches to Deep Networks – Capsules (Hinton), HTM (Numenta), Sparsey (Neurithmic Systems) and RCN (Vicarious)
What’s AlphaGo? AlphaGo is a system that can play Go at least as well as the best humans. Go was widely cited as the hardest (and only remaining) game at which humans could beat machines, so this is a big deal. AlphaGo has just defeated a top-ranked human expert. AlphaGo… Read More »What’s after AlphaGo?
This week we have a couple of interesting links to share. From our experiments with generative hierarchical models, we claimed that the model produced by feed-forward processing should not have loops. Now we have discovered a paper by Bengio et al titled “Towards biologically plausible deep learning” [1] that supports this… Read More »Some interesting finds: Acyclic hierarchical modelling and sequence unfolding
This month’s reading list continues with a subtheme on recurrent neural networks, and in particular Long Short Term Memory (LSTM). First here’s an interesting report on a panel discussion about the future of Deep Learning at the International Conference on Machine Learning (ICML), 2015: http://deeplearning.net/2015/07/13/a-brief-summary-of-the-panel-discussion-at-dl-workshop-icml-2015/ Participants included Yoshua Bengio (University… Read More »Reading list – July 2015
John Lisman, “The Challenge of Understanding the Brain: Where We Stand in 2015“, Neuron, 2015 For many in ML and AI, biological knowledge is focussed on cortex. This paper gives an excellent broad overview of current biological understanding of intelligence. Sebastian Billaudelle and Subutai Ahmad, “Porting HTM Models to the Heidelberg Neuromorphic… Read More »Reading List – May 2015
DeepMind, based on a deep learning architecture made a big impact in the tech industry. Deep learning continues to gain traction in the machine learning community and in the mainstream, both in terms of public awareness of its capabilities as well as through commercialisation. This article by Simon Winder provides… Read More »Deep learning, a history in context
Browsing the NUPIC Theory mailing list, I came across a post by Fergal Byrne on the differences and similarities between Deep Learning and MPF/HTM. It’s a great background into some of the pros and cons of each. Given the popularity and demonstrated success of Deep Learning methods it’s good to understand… Read More »A Unifying View of Deep Networks and Hierarchical Temporal Memory