Saturday, April 30, 2011

Neural Networks making a come-back?

Five years ago I ran some queries on Google Scholar to see trends on the number of papers that mention particular phrase. The number of hits for each year was divided by the number of hits for "machine learning". Back then it looked like NN's started gaining in popularity with invention of back-propagation in 1980's, peaked in 1993 and went downhill from there.



Since then, there's been several major NN developments, involving deep learning and probabilistically founded versions so I decided to update the trend. I couldn't find a copy of scholar scraper script anymore, luckily Konstantin Tretjakov has maintained a working version and reran the query for me.



It looks like downward trend in 2000's was misleading because not all papers from that period have made it into index yet, and the actual recent trend is exponential growth!

One example of this "third wave" of Neural Network research is unsupervised feature learning. Here's what you get if you train a sparse auto-encoder on some natural scene images



What you get is pretty much a set of Gabor filters, but the cool thing is that you get them from your neural network rather than image processing expert

42 comments:

Ted said...

"Don't call it a come-back"
http://en.wikipedia.org/wiki/Mama_Said_Knock_You_Out_%28song%29

Sid said...

Maybe you should mention a reference to Hintons 2006 work and how it affects all deep learning architectures?

Yaroslav said...

I would if I knew, what's so great about Hinton's 2006 paper?

ivan said...

Hinton's "reduced Boltzmann machine" is a deep neural network -- many layers. For some reason, a lot of the old NN literature only considered one or two level deep NNs. More layers = better.

I would recommend his tech talk:
http://www.youtube.com/watch?v=AyzOUbkUf3M

Mihail Sirotenko said...

The problem is that backprop can't efficiently learn multilayer networks to result in deep architecture. Network can have 10 layers but it's still shallow. So recent advances like RBMs or sparce autoencoders making this "come back".

tweak said...

As far as I understood, Hinton's 2006 paper was what started the whole autoencoder and pretraining wave...

Igor said...

Maybe it's a question that ought to be asked to either:
- the stats or metaoptimize Q&A
- Andrew

According to the wikipedia, the auto-encoders algorithm seem to help speed up the back propagation step (http://en.wikipedia.org/wiki/Auto-encoder ).

I am curious too.

Cheers,

Igor.

Gokul Ravi said...

nice blog
android training in bangalore
ios training in bangalore
machine learning online training

Gokul Ravi said...

useful blog
python interview questions
cognos interview questions
perl interview questions
vlsi interview questions
web api interview questions
msbi interview questions

Gokul Ravi said...

laravel interview questions
aem interview questions
salesforce interview questions
oops abab interview questions
itil interview questions
informatica interview questions
extjs interview questions

Gokul Ravi said...

sap bi interview questions
hive interview questions
seo interview questions
as400 interview questions
wordpress interview questions
accounting interview questions
basic accounting and financial interview questions

Neena Joshi said...

What a fantastic read on Hadoop.This has helped me understand a lot in Hadoop course. Please keep sharing similar write ups on Hadoop.Guys if you are keen to knw more on Hadoop, must check this wonderful Hadoop tutorial and i'm sure you will enjoy learning on Hadoop training.https://www.youtube.com/watch?v=ApR6pHN-m6M

akhila priya said...

Nice information thank you,if you want more information please visit our link MSBI online course

Radha Sai said...

Nice post.Keep updating Artificial intelligence Online Trining

Alamgir Hossain said...

This neural networking side should have been there so that the overall factors lying to this could be into our mind. https://www.academicposter.net/faq-how-to-make-an-academic-poster/ to see more about the writing tips.

Charles Kee said...

Who would ever thought that the Mary Wood will go this long. This is really some lovely news about her that she can do better in the acting profession in her way. https://radiology.residencypersonalstatements.net/radiology-personal-statement-writing/ & it will give you some amazing idea and very effective for the papers writing.

Arthur Mendoza said...

It is really a good decision to know about the register number in really easy way which let the people in relief that there had no any tough situation will face in later time. https://www.writemyessays.org/how-to-write-a-3-page-essay-fast/ for the students that is very helpful for the writing services.

Priya Rajesh said...

Excellent blog.
Machine Learning Training course

Geetha Devi said...

Great post. I absolutely love to find unique places like this. It really looks super creepy though!! Best Machine Learning Training in Chennai | best machine learning institute in chennai | Machine Learning course in chennai

Lucy T. Patterson said...

there has so many reason behind the love of this philosophy subject as mentioned http://www.applicationessayprompts.com/answers-to-college-essay-prompts/help-with-writing-a-college-application-essay/
/ as it gives you that chance to know about people's mind.

robert said...

If they are really come back to the exact point http://www.conferenceresearchpaper.com/word-to-speech-converter/ and situation then none could overtake them like they was in those past time history. Check about them in here.

bigclasses 04 said...

Thank you .its a nice blog.
click here:
Data Science Online Traning

Anonymous said...

Selenium is one of the most popular automated testing tool used to automate various types of applications. Selenium is a package of several testing tools designed in a way for to support and encourage automation testing of functional aspects of web-based applications and a wide range of browsers and platforms and for the same reason, it is referred to as a Suite.

Selenium Interview Questions and Answers
Javascript Interview Questions
Human Resource (HR) Interview Questions

jefrin adams said...

Very nice to read keep on posting
Machine learning training in chennai

nivedhitha reddy said...

very nice article Leadingmachine learning training in ameerpet

VRITPROFESSIONALS said...

Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
Thanks & Regards,
VRIT Professionals,
No.1 Leading Web Designing Training Institute In Chennai.

And also those who are looking for
Web Designing Training Institute in Chennai
SEO Training Institute in Chennai
Photoshop Training Institute in Chennai
PHP & Mysql Training Institute in Chennai
Android Training Institute in Chennai

jefrin adams said...

Informative post for machine learning
data science training institute in chennai

basha said...

Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative.
oracle fusion financials classroom training
Workday HCM Online Training
Oracle Fusion Financials Online Training
Oracle Fusion HCM Online Training
Oracle Fusion SCM Online Training
Oracle Fusion HCM Classroom Training

lekha mathan said...

Such an excellent and interesting blog, do post like this more with more information, This was very useful, Thank you.
Aviation Academy in Chennai
Air hostess training in Chennai
Airport management courses in Chennai
Ground staff training in Chennai
aviation training in Chennai
cabin crew course in Chennai
diploma in airline and airport management in Chennai
airport ground staff training in Chennai

Ramya Krishnan said...

This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

Regards,
Ramya

Azure Training in Chennai
Azure Training Center in Chennai
Best Azure Training in Chennai
Azure Devops Training in Chenna
Azure Training Institute in Chennai
Azure Training in Chennai OMR
Azure Training in Chennai Velachery
Azure Online Training
Azure Training in Chennai Credo Systemz
DevOps Training in Chennai Credo Systemz

Soumitasai said...

It's really a nice experience to read your post. Thank you for sharing this useful information

big data hadoop training cost in chennai | hadoop training in Chennai | best bigdata hadoop training in chennai | best hadoop certification in Chennai

IT Tutorials said...

Really useful information. Thank you so much for sharing.It will help everyone.Keep Post. RPA training in chennai | RPA training in Chennai with placement | UiPath training in Chennai | UiPath certification in Chennai with cost

Realtekh Consulting said...



Nice article, interesting to read…
Thanks for sharing the useful information

erp in chennai
erp implementation in chennai
erp software solutions in chennai
erp software development company in chennai
Machine Maintanance Software in chennai

Nisha San said...

Hey, would you mind if I share your blog with my twitter group? There’s a lot of folks that I think would enjoy your content. Please let me know. Thank you.
machine learning Training in Chennai | machine learning Training institute in chennai | Best machine learning Training in chennai

Unknown said...

THANKS FOR THE INFORMATION....
Digital Marketing Internship Program in BangaloreDigital Marketing Internship Program in Bangalore

draj said...


Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

sabaribalaji said...

nice explanation, thanks for sharing it is very informative
top 100 machine learning interview questions
top 100 machine learning interview questions and answers
Machine learning interview questions
Machine learning job interview questions
Machine learning interview questions techtutorial

sabaribalaji said...
This comment has been removed by the author.
sabaribalaji said...

nice blog thanks for sharing
Machine learning job interview questions and answers
Machine learning interview questions and answers online
Machine learning interview questions and answers for freshers
interview question for machine learning
machine learning interview questions and answers

Malaivel Siddha Hospital said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
Kindly visit us @
Top HIV Hospital in India | Best AIDS Hospital in India
HIV AIDS Treatment in Bangalore | HIV Specialist in Bangalore
Best HIV Doctor in India
Cure best blood cancer treatment in Tamilnadu

Rigid Box said...

Excellent Blog. I really want to admire the quality of this post. I like the way of your presentation of ideas, views and valuable content. No doubt you are doing great work. I’ll be waiting for your next post. Thanks .Keep it up!
Kindly visit us @
Luxury Boxes
Premium Packaging
Luxury Candles Box
Earphone Packaging Box
Wireless Headphone Box
Innovative Packaging Boxes
Wedding gift box
Leather Bag Packaging Box
Cosmetics Packaging Box
Luxury Chocolate Boxes

KIT said...

Nice blog, it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
Kindly visit us @
100% Job Placement
Best Colleges for Computer Engineering
Biomedical Engineering Colleges in Coimbatore
Best Biotechnology Colleges in Tamilnadu
Biotechnology Colleges in Coimbatore
Biotechnology Courses in Coimbatore
Best MCA Colleges in Tamilnadu
Best MBA Colleges in Coimbatore
Engineering Courses in Tamilnadu
Engg Colleges in Coimbatore