Saturday, April 30, 2011

Neural Networks making a come-back?

Five years ago I ran some queries on Google Scholar to see trends on the number of papers that mention particular phrase. The number of hits for each year was divided by the number of hits for "machine learning". Back then it looked like NN's started gaining in popularity with invention of back-propagation in 1980's, peaked in 1993 and went downhill from there.



Since then, there's been several major NN developments, involving deep learning and probabilistically founded versions so I decided to update the trend. I couldn't find a copy of scholar scraper script anymore, luckily Konstantin Tretjakov has maintained a working version and reran the query for me.



It looks like downward trend in 2000's was misleading because not all papers from that period have made it into index yet, and the actual recent trend is exponential growth!

One example of this "third wave" of Neural Network research is unsupervised feature learning. Here's what you get if you train a sparse auto-encoder on some natural scene images



What you get is pretty much a set of Gabor filters, but the cool thing is that you get them from your neural network rather than image processing expert

76 comments:

Ted said...

"Don't call it a come-back"
http://en.wikipedia.org/wiki/Mama_Said_Knock_You_Out_%28song%29

Sid said...

Maybe you should mention a reference to Hintons 2006 work and how it affects all deep learning architectures?

Yaroslav Bulatov said...

I would if I knew, what's so great about Hinton's 2006 paper?

ivan said...

Hinton's "reduced Boltzmann machine" is a deep neural network -- many layers. For some reason, a lot of the old NN literature only considered one or two level deep NNs. More layers = better.

I would recommend his tech talk:
http://www.youtube.com/watch?v=AyzOUbkUf3M

Unknown said...

The problem is that backprop can't efficiently learn multilayer networks to result in deep architecture. Network can have 10 layers but it's still shallow. So recent advances like RBMs or sparce autoencoders making this "come back".

Andreas Mueller said...

As far as I understood, Hinton's 2006 paper was what started the whole autoencoder and pretraining wave...

Igor said...

Maybe it's a question that ought to be asked to either:
- the stats or metaoptimize Q&A
- Andrew

According to the wikipedia, the auto-encoders algorithm seem to help speed up the back propagation step (http://en.wikipedia.org/wiki/Auto-encoder ).

I am curious too.

Cheers,

Igor.

Unknown said...

nice blog
android training in bangalore
ios training in bangalore
machine learning online training

Unknown said...

useful blog
python interview questions
cognos interview questions
perl interview questions
vlsi interview questions
web api interview questions
msbi interview questions

Unknown said...

laravel interview questions
aem interview questions
salesforce interview questions
oops abab interview questions
itil interview questions
informatica interview questions
extjs interview questions

Unknown said...

sap bi interview questions
hive interview questions
seo interview questions
as400 interview questions
wordpress interview questions
accounting interview questions
basic accounting and financial interview questions

Unknown said...

What a fantastic read on Hadoop.This has helped me understand a lot in Hadoop course. Please keep sharing similar write ups on Hadoop.Guys if you are keen to knw more on Hadoop, must check this wonderful Hadoop tutorial and i'm sure you will enjoy learning on Hadoop training.https://www.youtube.com/watch?v=ApR6pHN-m6M

akhilapriya404 said...

Nice information thank you,if you want more information please visit our link MSBI online course

radha said...

Nice post.Keep updating Artificial intelligence Online Trining

Abir said...

This neural networking side should have been there so that the overall factors lying to this could be into our mind. https://www.academicposter.net/faq-how-to-make-an-academic-poster/ to see more about the writing tips.

Charles Kee said...

Who would ever thought that the Mary Wood will go this long. This is really some lovely news about her that she can do better in the acting profession in her way. https://radiology.residencypersonalstatements.net/radiology-personal-statement-writing/ & it will give you some amazing idea and very effective for the papers writing.

Unknown said...

It is really a good decision to know about the register number in really easy way which let the people in relief that there had no any tough situation will face in later time. https://www.writemyessays.org/how-to-write-a-3-page-essay-fast/ for the students that is very helpful for the writing services.

priya rajesh said...

Excellent blog.
Machine Learning Training course

Unknown said...

Great post. I absolutely love to find unique places like this. It really looks super creepy though!! Best Machine Learning Training in Chennai | best machine learning institute in chennai | Machine Learning course in chennai

Lucy T. Patterson said...

there has so many reason behind the love of this philosophy subject as mentioned http://www.applicationessayprompts.com/answers-to-college-essay-prompts/help-with-writing-a-college-application-essay/
/ as it gives you that chance to know about people's mind.

robert said...

If they are really come back to the exact point http://www.conferenceresearchpaper.com/word-to-speech-converter/ and situation then none could overtake them like they was in those past time history. Check about them in here.

Unknown said...

Thank you .its a nice blog.
click here:
Data Science Online Traning

Anonymous said...

Selenium is one of the most popular automated testing tool used to automate various types of applications. Selenium is a package of several testing tools designed in a way for to support and encourage automation testing of functional aspects of web-based applications and a wide range of browsers and platforms and for the same reason, it is referred to as a Suite.

Selenium Interview Questions and Answers
Javascript Interview Questions
Human Resource (HR) Interview Questions

jefrin said...

Very nice to read keep on posting
Machine learning training in chennai

nivedhitha said...

very nice article Leadingmachine learning training in ameerpet

VRITPROFESSIONALS said...

Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
Thanks & Regards,
VRIT Professionals,
No.1 Leading Web Designing Training Institute In Chennai.

And also those who are looking for
Web Designing Training Institute in Chennai
SEO Training Institute in Chennai
Photoshop Training Institute in Chennai
PHP & Mysql Training Institute in Chennai
Android Training Institute in Chennai

jefrin said...

Informative post for machine learning
data science training institute in chennai

basha said...

Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative.
oracle fusion financials classroom training
Workday HCM Online Training
Oracle Fusion Financials Online Training
Oracle Fusion HCM Online Training
Oracle Fusion SCM Online Training
Oracle Fusion HCM Classroom Training

lekha mathan said...

Such an excellent and interesting blog, do post like this more with more information, This was very useful, Thank you.
Aviation Academy in Chennai
Air hostess training in Chennai
Airport management courses in Chennai
Ground staff training in Chennai
aviation training in Chennai
cabin crew course in Chennai
diploma in airline and airport management in Chennai
airport ground staff training in Chennai

Ramya Krishnan said...

This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

Regards,
Ramya

Azure Training in Chennai
Azure Training Center in Chennai
Best Azure Training in Chennai
Azure Devops Training in Chenna
Azure Training Institute in Chennai
Azure Training in Chennai OMR
Azure Training in Chennai Velachery
Azure Online Training
Azure Training in Chennai Credo Systemz
DevOps Training in Chennai Credo Systemz

Soumitasai said...

It's really a nice experience to read your post. Thank you for sharing this useful information

big data hadoop training cost in chennai | hadoop training in Chennai | best bigdata hadoop training in chennai | best hadoop certification in Chennai

IT Tutorials said...

Really useful information. Thank you so much for sharing.It will help everyone.Keep Post. RPA training in chennai | RPA training in Chennai with placement | UiPath training in Chennai | UiPath certification in Chennai with cost

Realtek Consulting said...



Nice article, interesting to read…
Thanks for sharing the useful information

erp in chennai
erp implementation in chennai
erp software solutions in chennai
erp software development company in chennai
Machine Maintanance Software in chennai

Nisha San said...

Hey, would you mind if I share your blog with my twitter group? There’s a lot of folks that I think would enjoy your content. Please let me know. Thank you.
machine learning Training in Chennai | machine learning Training institute in chennai | Best machine learning Training in chennai

remo said...

THANKS FOR THE INFORMATION....
Digital Marketing Internship Program in BangaloreDigital Marketing Internship Program in Bangalore

draj said...


Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

sabaribalaji said...

nice explanation, thanks for sharing it is very informative
top 100 machine learning interview questions
top 100 machine learning interview questions and answers
Machine learning interview questions
Machine learning job interview questions
Machine learning interview questions techtutorial

sabaribalaji said...
This comment has been removed by the author.
sabaribalaji said...

nice blog thanks for sharing
Machine learning job interview questions and answers
Machine learning interview questions and answers online
Machine learning interview questions and answers for freshers
interview question for machine learning
machine learning interview questions and answers

Malaivel Siddha Hospital said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
Kindly visit us @
Top HIV Hospital in India | Best AIDS Hospital in India
HIV AIDS Treatment in Bangalore | HIV Specialist in Bangalore
Best HIV Doctor in India
Cure best blood cancer treatment in Tamilnadu

Rigid Box said...

Excellent Blog. I really want to admire the quality of this post. I like the way of your presentation of ideas, views and valuable content. No doubt you are doing great work. I’ll be waiting for your next post. Thanks .Keep it up!
Kindly visit us @
Luxury Boxes
Premium Packaging
Luxury Candles Box
Earphone Packaging Box
Wireless Headphone Box
Innovative Packaging Boxes
Wedding gift box
Leather Bag Packaging Box
Cosmetics Packaging Box
Luxury Chocolate Boxes

KIT said...

Nice blog, it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
Kindly visit us @
100% Job Placement
Best Colleges for Computer Engineering
Biomedical Engineering Colleges in Coimbatore
Best Biotechnology Colleges in Tamilnadu
Biotechnology Colleges in Coimbatore
Biotechnology Courses in Coimbatore
Best MCA Colleges in Tamilnadu
Best MBA Colleges in Coimbatore
Engineering Courses in Tamilnadu
Engg Colleges in Coimbatore

sabaribalaji said...

thanks for this informative article it is very useful
Machine learning taining in chennai
artificial intelligence and machine learning course in chennai
best machine learning training institute
top institutes for machine learning in chennai
machine learning course training institute in chennai
machine learning certification
machine learning training institutes
best institute to learn machine learning in chennai
machine learning course training
machine learning with r training in chennai

Raj Sharma said...

Good Post. I like your blog. Thanks for Sharing
Machine Learning Training in Noida

Venkatesh CS said...

Excellent Blog. Thank you so much for sharing.
best react js training in chennai
react js training in Chennai
react js workshop in Chennai
react js courses in Chennai
react js training institute in Chennai
reactjs training Chennai
react js online training
react js online training india
react js course content
react js training courses
react js course syllabus
react js training
react js certification in chennai
best react js training

Aditi Digital Solutions said...

good article about data science has given it is very nice thank you for sharing.
Data Science Training in Hyderabad

LifeVoxel.AI said...

LifeVoxel.AI has developed a Interactive Streaming and AI Platform for medical imaging using GPU clusters cloud computing. It is a leap in cloud technology platform in medical imaging that encompasses use cases in visualization, AI, image management and workflow. It’s approach is unique that it has been granted 12 International patents. LifeVoxel.AI’s platform is certified for HIPAA compliancy. The platform was granted an FDA 510K approval for use in diagnostic interpretation of medical images.

Interactive Streaming AI Platform RIS PACS

Venkatesh CS said...

Thanks for sharing valuable information.
Digital Marketing training Course in chennai
digital marketing training institute in chennai
digital marketing training in Chennai
digital marketing course in Chennai
digital marketing course training in omr
digital marketing certification in omr
digital marketing course training in velachery
digital marketing training center in chennai
digital marketing courses with placement in chennai
digital marketing certification in chennai
digital marketing institute in Chennai
digital marketing certification course in Chennai
digital marketing course training in Chennai
Digital Marketing course in Chennai with placement
digital marketing courses in chennai

KIT said...


A very inspiring blog your article is so convincing that I never stop myself to say something about it.

Jackie Co Kad said...

Great Article
Final Year Project Domains for CSE
Final Year Project Centers in Chennai





JavaScript Training in Chennai
JavaScript Training in Chennai

Anonymous said...

Thanks for sharing such a great blog Keep posting.. 
Machine Learning Training in Delhi

Malaivel Siddha Hospital said...

Wonderful post, This article have helped greatly continue writing ..

Malaivel Siddha Hospital said...


I´ve been thinking of starting a blog on this subject myself .....

Kanye Co Jamila said...


Great Article
IEEE final year projects on machine learning


JavaScript Training in Chennai

Final Year Project Centers in Chennai



JavaScript Training in Chennai

KIT said...

A very interesting blog....

Itools Mobile Service said...

Thanks for sharing information. I really appreciate it.

Savanah Eden said...

We help you to get high traffic for your websites using latest ethics,visit us
Linkedin Marketing company chennai
Erp software development company in chennai
seo company in chennai
Web portal development company in chennai

LifeVoxel.AI said...

LifeVoxel.AI platform helps imaging diagnostic centers and hospitals to save up to 50%+ over conventional RIS PACS with higher functionality. LifeVoxel.AI is the fastest RIS PACS available globally and have unimaginable capabilities of centralized PACS across all your network of Imaging Centers to single window HUB.

RIS PACS
RIS PACS software

Malaivel Siddha Hospital said...

Thanks for sharing...
Very good Keep it up.


Kanagamani Hospital said...

Thanks for the post. I´ve been thinking of starting a blog on this subject myself.
Keep sharing.

Shanthi Cabs said...

Nice blog, it’s so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.

Garrick Co Ida said...

The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. Machine Learning Final Year Projects In case you will succeed, you have to begin building machine learning projects in the near future.

Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

KIT said...

A very inspiring blog your article is so convincing that I never stop myself to say something about it.

Malaivel Siddha Hospital said...

Wonderful post, This article have helped greatly continue writing ..

Malaivel Siddha Hospital said...

I´ve been thinking of starting a blog on this subject myself .....

KIT said...

A very interesting blog....

Malaivel Siddha Hospital said...

Thanks for sharing...
Very good Keep it up.

Shanthi Cabs said...

Nice blog, it’s so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.

Daily Health Tips said...

Hello Admin!

Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuffs like this. If you are looking for the Advertising Agency in Chennai | Printing in Chennai , Visit Inoventic Creative Agency Today..

Shanthi Cabs said...

Keep sharing the post like this...

jeetu said...

Web Designer in Noida
Best Website Development service In Noida
Website Designing service In Noida
Best digital marketing service In Noida
Best digital marketing Company in Noida
Indian Bookmarking list
Indian Bookmarking list
India Classified Submission List
Indian Classified List

SS Vermicompost Industry said...

Thanks for sharing such a great blog
Vermicompost Manufacturers | Best Vermicompost in chennai

Shanthi Cabs said...

Nice blog, it’s so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
Tours and Travels in Madurai | Best tour operators in Madurai
Best travel agency in Madurai | Best Travels in Madurai



tjproductstech said...


Hi, Very nice article. I hope you will publish again such type of post. Thank you!
Corporate gifts ideas | Corporate gifts
Corporate gifts singapore | Corporate gifts in singapore
Promotional gifts singapore | Corporate gifts wholesale Singapore

Agalya said...

Nice Blog..thanks for sharing this information..

Best Data Science Training in Chennai
Top 5 Data Science Training in Chennai
Data Science training Course Content
Data Science Training in Velachery
Data Science Training in omr
Data Science Training in vadapalani
Data Science Training in Chennai
Data Science Courses in Chennai
Data Science Training Institute in Chennai
Data Science online course
Data Science with python training in chennai

SS Vermicompost Industry said...


Thanks for sharing such a great blog
Vermicompost manufacturers in Tamilnadu | Vermicompost in Tamilnadu
Vermicompost Manufacturers | Vermicompost Suppliers
Vermicompost in Coimbatore | Vermicompost manufacturers in Chennai
Vermicompost in chennai | Best Vermicompost in chennai
.