Saturday, April 30, 2011

Neural Networks making a come-back?

Five years ago I ran some queries on Google Scholar to see trends on the number of papers that mention particular phrase. The number of hits for each year was divided by the number of hits for "machine learning". Back then it looked like NN's started gaining in popularity with invention of back-propagation in 1980's, peaked in 1993 and went downhill from there.



Since then, there's been several major NN developments, involving deep learning and probabilistically founded versions so I decided to update the trend. I couldn't find a copy of scholar scraper script anymore, luckily Konstantin Tretjakov has maintained a working version and reran the query for me.



It looks like downward trend in 2000's was misleading because not all papers from that period have made it into index yet, and the actual recent trend is exponential growth!

One example of this "third wave" of Neural Network research is unsupervised feature learning. Here's what you get if you train a sparse auto-encoder on some natural scene images



What you get is pretty much a set of Gabor filters, but the cool thing is that you get them from your neural network rather than image processing expert

23 comments:

Ted said...

"Don't call it a come-back"
http://en.wikipedia.org/wiki/Mama_Said_Knock_You_Out_%28song%29

Sid said...

Maybe you should mention a reference to Hintons 2006 work and how it affects all deep learning architectures?

Yaroslav said...

I would if I knew, what's so great about Hinton's 2006 paper?

ivan said...

Hinton's "reduced Boltzmann machine" is a deep neural network -- many layers. For some reason, a lot of the old NN literature only considered one or two level deep NNs. More layers = better.

I would recommend his tech talk:
http://www.youtube.com/watch?v=AyzOUbkUf3M

Mihail Sirotenko said...

The problem is that backprop can't efficiently learn multilayer networks to result in deep architecture. Network can have 10 layers but it's still shallow. So recent advances like RBMs or sparce autoencoders making this "come back".

tweak said...

As far as I understood, Hinton's 2006 paper was what started the whole autoencoder and pretraining wave...

Igor said...

Maybe it's a question that ought to be asked to either:
- the stats or metaoptimize Q&A
- Andrew

According to the wikipedia, the auto-encoders algorithm seem to help speed up the back propagation step (http://en.wikipedia.org/wiki/Auto-encoder ).

I am curious too.

Cheers,

Igor.

Gokul Ravi said...

nice blog
android training in bangalore
ios training in bangalore
machine learning online training

Gokul Ravi said...

useful blog
python interview questions
cognos interview questions
perl interview questions
vlsi interview questions
web api interview questions
msbi interview questions

Gokul Ravi said...

laravel interview questions
aem interview questions
salesforce interview questions
oops abab interview questions
itil interview questions
informatica interview questions
extjs interview questions

Gokul Ravi said...

sap bi interview questions
hive interview questions
seo interview questions
as400 interview questions
wordpress interview questions
accounting interview questions
basic accounting and financial interview questions

Neena Joshi said...

What a fantastic read on Hadoop.This has helped me understand a lot in Hadoop course. Please keep sharing similar write ups on Hadoop.Guys if you are keen to knw more on Hadoop, must check this wonderful Hadoop tutorial and i'm sure you will enjoy learning on Hadoop training.https://www.youtube.com/watch?v=ApR6pHN-m6M

akhila priya said...

Nice information thank you,if you want more information please visit our link MSBI online course

Radha Sai said...

Nice post.Keep updating Artificial intelligence Online Trining

Alamgir Hossain said...

This neural networking side should have been there so that the overall factors lying to this could be into our mind. https://www.academicposter.net/faq-how-to-make-an-academic-poster/ to see more about the writing tips.

Charles Kee said...

Who would ever thought that the Mary Wood will go this long. This is really some lovely news about her that she can do better in the acting profession in her way. https://radiology.residencypersonalstatements.net/radiology-personal-statement-writing/ & it will give you some amazing idea and very effective for the papers writing.

Arthur Mendoza said...

It is really a good decision to know about the register number in really easy way which let the people in relief that there had no any tough situation will face in later time. https://www.writemyessays.org/how-to-write-a-3-page-essay-fast/ for the students that is very helpful for the writing services.

Priya Rajesh said...

Excellent blog.
Machine Learning Training course

Geetha Devi said...

Great post. I absolutely love to find unique places like this. It really looks super creepy though!! Best Machine Learning Training in Chennai | best machine learning institute in chennai | Machine Learning course in chennai

Lucy T. Patterson said...

there has so many reason behind the love of this philosophy subject as mentioned http://www.applicationessayprompts.com/answers-to-college-essay-prompts/help-with-writing-a-college-application-essay/
/ as it gives you that chance to know about people's mind.

robert said...

If they are really come back to the exact point http://www.conferenceresearchpaper.com/word-to-speech-converter/ and situation then none could overtake them like they was in those past time history. Check about them in here.

bigclasses 04 said...

Thank you .its a nice blog.
click here:
Data Science Online Traning

Anonymous said...

Selenium is one of the most popular automated testing tool used to automate various types of applications. Selenium is a package of several testing tools designed in a way for to support and encourage automation testing of functional aspects of web-based applications and a wide range of browsers and platforms and for the same reason, it is referred to as a Suite.

Selenium Interview Questions and Answers
Javascript Interview Questions
Human Resource (HR) Interview Questions