Sunday, September 05, 2010

MaxEnt or Bayesian?

Foundations of probabilistic inference is often a subject of much disagreement, with some leading Bayesian sometimes going as far as to say that MaxEnt method doesn't make any sense, and the MaxEnt camp picking on the issue of subjectivity.

The way I see it, MaxEnt and Bayesian approaches are just different ways of using external information to pick a probability distribution.

With Bayesian approach, you are given data and a prior. Bayesian inference then comes out as a natural extension of logic to probabilities. See Ch.1 of Cox's Algebra of Probable Inference for full axiomatic derivation.

With MaxEnt approach, you are given a set of valid probability distributions. You can then justify MaxEnt axiomatically but personally I like the motivation given in Jaynes book, Probability Theory: The Logic of Science (in particular Ch.11).

Roughly speaking, it goes as follows. Suppose we have a distibution $p$ over $d$ outcomes. We say that a sequence of $n$ observations is explained by distribution $p$ if relative counts of outcomes in this sequence match $p$ exactly. The number of sequences matched by $p$ is the following quantity

$$c(p)=\frac{n!}{(np_1)!(np_2)!\cdots (np_d)!}$$

If we have a set of valid distributions $p$, it then makes sense to pick the one with highest $c(p)$ because it'll fit the greatest number of possible future datasets. Approximating $n!$ by $n^n$ we get the following

$$\frac{1}{n}\log c(p)\approx \frac{1}{n}\log \frac{n^n}{(np_1)^{np_1}(np_2)^{np_2}\cdots (np_d)^{np_d}}=H(p)$$

Hence picking distribution with highest H(p) usually means picking the distribution that will be able to explain the most possible observation sequences. Because it's an approximation, there can be exceptions, for instance distributions $(\frac{12}{18},\frac{3}{18},\frac{3}{18})$ and $(\frac{9}{18},\frac{8}{18},\frac{1}{18})$ have equal entropy, but the latter explains 18% more length 18 sequences.

To carry out the inference, Bayesian approach needs to start with a prior, whereas MaxEnt approach needs to start with a set of valid distributions. In practice people often use constant function for prior, and "distributions for which expected feature counts match observed feature counts" for the valid set of distributions.

Note that when dealing with continuous distributions, result of MaxEnt inference depends on the choice of measure over our set of distributions, so I think it's hard to justify in continuous case.

Here are some papers I scanned a while back on foundations of Bayesian and MaxEnt approaches.

45 comments:

Yaroslav Bulatov said...

Just came across this related post by Anand, apparently Feller was critical of Bayesian approach

http://ergodicity.net/2010/08/17/fellers-anti-bayesianism/

remo said...

THANKS FOR THE INFORMATION...
<a href="http://www.chloros.in/digital-marketing-internship.htmlhttp://www.chloros.in/digital-marketing-internship.html>Digital Marketing Internship Program in Bangalore</a>

draj said...



Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

Best Data Science Course Training in Hyderabad-360DigiTMG said...

I have recently started to read this blog, the info you provide about Machine Learning on this post has helped me greatly. Thanks for all of your time & work.Learn best
Data Science Course in Hyderabad using python and R programming

Best Business Analytics Course in Hyderabad said...

I feel really happy to have seen your post and look forward to so many more entertaining times reading here. Thanks once more for all the details.Learn best Business Analytics Course in Hyderabad

innomatics research labs said...


Data Science Course Training in Hyderabad


Data Science Training in Hyderabad, Data Science Course in Hyderabad, Data Science Training, Data Science Course, Data Science Institute, Data Science Certification




Data Science Course in Hyderabad

Data Science Training in Hyderabad
Data Science Course Training in Hyderabad


Advanced-Data science training with Free Internship & 100% Placement Assistance

About the Data Science course Training in Hyderabad

Data is everywhere, which is growing exponentially globally, and this may still grow at an accelerating rate for the foreseeable future. Businesses generate massive amounts of data within the type of blogs, messages, transaction documents, mobile device data, social media, etc. By using this data effectively, a firm can create vital value and grow their economy by enhancing productivity, increasing efficiency, and delivering more value to consumers.

Data Science helps in combining the disruption into categories and communicating their potential, which allows data and analytics leaders to drive better results. Top businesses thought there is a necessity to research the data for significant benefits. They use the insights from data for the advantage of users.

Human deciding is becoming increasingly inadequate to pander to a never-ending expansion of the data . However, Data Science and Machine Learning are excelling in solving highly complex data-rich problems. To think beyond the human brain and maintain the balance with the knowledge that's evolved, disrupted, and being employed the sectors altogether, data scientists foster new methodologies. Data scientists must try 'big data expeditions' to explore the data for previously undiscovered value - the first common application of data science. Typical applications include marketing segmentation, advertising, tweaking dynamic pricing models, or banks finding risks and adjusting the financial risk models.

What are the Tools utilized in Data Science?

prathyusha said...

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science institutes in Hyderabad

360digitmg said...

I think about it is most required for making more on this get engaged
Best Data Science Courses in Hyderabad

prathyusha said...

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science courses in Hyderabad

Maneesha said...

All things considered I read it yesterday however I had a few considerations about it and today I needed to peruse it again on the grounds that it is very elegantly composed.
data scientist training

Mallela said...

Thanks for posting the best information and the blog is very informative.Data science course in Faridabad

traininginstitute said...

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
digital marketing courses in hyderabad with placement

Mallela said...

Thanks for posting the best information and the blog is very important.data science institutes in hyderabad

Deekshitha said...

Informative blog
data analytics courses in hyderabad

Anonymous said...

Nice blog post,
Digital Marketing Trainer in Hyderabad

traininginstitute said...

I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place
digital marketing courses in hyderabad with placement


Priya Rathod said...

An amazing web journal I visit this blog, it's unbelievably wonderful.
DevOps Training in Hyderabad
DevOps Course in Hyderabad

Data Science Course in Bhilai - 360DigiTMG said...

Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

data science course in varanasi

Unknown said...

Fantastic article I ought to say and thanks to the info. Instruction is absolutely a sticky topic. But remains one of the top issues of the time. I love your article and look forward to more.
Data Science Course in Bangalore

DataScienceEXPO said...

Thank you quite much for discussing this type of helpful informative article. Will certainly stored and reevaluate your Website.
Data Analytics Course in Bangalore

DataBank said...

Nice to be seeing your site once again, it's been weeks for me. This article which ive been waited for so long. I need this guide to complete my mission inside the school, and it's same issue together along with your essay. Thanks, pleasant share.
Data Science training in Bangalore

DataCharge said...

Very wonderful informative article. I appreciated looking at your article. Very wonderful reveal. I would like to twit this on my followers. Many thanks! .
Data Analytics training in Bangalore

Priya Rathod said...

Excellent blog and article. I enjoy looking at your posts. Thanks for sharing on the web. I am hoping the same best work from you in the future as well. Thank you for sharing great blogging with us!
Data Science Training in Hyderabad
Data Science Course in Hyderabad

Priya Rathod said...

Reading this article was very useful-- particularly for me, as it gave me some more insight into writing tips. I wanted to add that the author did a good job of explaining those points clearly. This is particularly helpful for new entrepreneurs because it shows how important it is to consider how everyone perceives your business from the outside. I'm looking forward reading other articles soon.
AWS Training in Hyderabad
AWS Course in Hyderabad

data scientist course said...

Excellent post.I want to thank you for this informative read, I really appreciate sharing this great post.Keep up your work
data scientist course in hyderabad

Mallela said...

Thanks for posting the best information and the blog is very good.data science course in Lucknow

data scientist course said...

Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share.
data scientist course in hyderabad

Ramesh Sampangi said...

Fantastic blog. Informative and Knowledgeable content. Keep sharing more blogs with us.
Machine Learning Course in Hyderabad

technologyforall said...
This comment has been removed by the author.
Ramesh Sampangi said...

Become a data science expert by joining AI Patasala’s Data Science Course Training in Hyderabad program, where data science aspirants can learn data science concepts with practical knowledge.
Data Science Course Training Institute in Hyderabad with Placements
Data Science Course Training in Hyderabad

faaantas said...




Hey friend, it is very well written article, thank you for the valuable and useful information you provide in this post. Keep up the good work! FYI, please check these depression, stress and anxiety related articles.
Federal Bank Signet Credit Card 2021 Review , The High Five Habit Free pdf Download , 10 lines about online classes in English

Deekshitha said...

Informative blog
data analytics course in jamshedpur

data science bangalore said...

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.business analytics course in rohtak

Unknown said...

I feel extremely appreciative that I read this. It is extremely useful and exceptionally enlightening and I truly took in a ton from it. data science course in surat

Unknown said...

I am truly getting a charge out of perusing your elegantly composed articles. It would seem that you burn through a ton of energy and time on your blog. I have bookmarked it and I am anticipating perusing new articles. Keep doing awesome. data science course in mysore

Project Management Course said...

You finished certain solid focuses there. I did a pursuit regarding the matter and discovered almost all people will concur with your blog.

traininginstitute said...

Nice knowledge gaining article. This post is really the best on this valuable topic.
data science training in malaysia

PMP Training in Malaysia said...

360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

data analytics course said...

All things considered I read it yesterday yet I had a few musings about it and today I needed to peruse it again in light of the fact that it is very elegantly composed. https://360digitmg.com/course/data-analytics-using-python-r

Unknown said...

I truly appreciate just perusing the entirety of your weblogs. Just needed to educate you that you have individuals like me who value your work. Unquestionably an extraordinary post. Caps off to you! The data that you have given is exceptionally useful.data science course in bhopal

traininginstitute said...

Excellent effort to make this blog more wonderful and attractive.
full stack developer course with placement

Unknown said...

it was a wonderful chance to visit this kind of site and I am happy to know. thank you so much for giving us a chance to have this opportunity.. data science course in mysore

Unknown said...

Very informative message! There is so much information here that can help any business start a successful social media campaign!
data science training in london

PMP Training in Malaysia said...

I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… PMP certification in malaysia

Mahil mithu said...

Well, I really appreciated for your great work. This topic submitted by you is helpful and keep sharing...
Separation Before Divorce
Cost of Legal Separation VS Divorce