Sunday, September 05, 2010

MaxEnt or Bayesian?

Foundations of probabilistic inference is often a subject of much disagreement, with some leading Bayesian sometimes going as far as to say that MaxEnt method doesn't make any sense, and the MaxEnt camp picking on the issue of subjectivity.

The way I see it, MaxEnt and Bayesian approaches are just different ways of using external information to pick a probability distribution.

With Bayesian approach, you are given data and a prior. Bayesian inference then comes out as a natural extension of logic to probabilities. See Ch.1 of Cox's Algebra of Probable Inference for full axiomatic derivation.

With MaxEnt approach, you are given a set of valid probability distributions. You can then justify MaxEnt axiomatically but personally I like the motivation given in Jaynes book, Probability Theory: The Logic of Science (in particular Ch.11).

Roughly speaking, it goes as follows. Suppose we have a distibution $p$ over $d$ outcomes. We say that a sequence of $n$ observations is explained by distribution $p$ if relative counts of outcomes in this sequence match $p$ exactly. The number of sequences matched by $p$ is the following quantity

$$c(p)=\frac{n!}{(np_1)!(np_2)!\cdots (np_d)!}$$

If we have a set of valid distributions $p$, it then makes sense to pick the one with highest $c(p)$ because it'll fit the greatest number of possible future datasets. Approximating $n!$ by $n^n$ we get the following

$$\frac{1}{n}\log c(p)\approx \frac{1}{n}\log \frac{n^n}{(np_1)^{np_1}(np_2)^{np_2}\cdots (np_d)^{np_d}}=H(p)$$

Hence picking distribution with highest H(p) usually means picking the distribution that will be able to explain the most possible observation sequences. Because it's an approximation, there can be exceptions, for instance distributions $(\frac{12}{18},\frac{3}{18},\frac{3}{18})$ and $(\frac{9}{18},\frac{8}{18},\frac{1}{18})$ have equal entropy, but the latter explains 18% more length 18 sequences.

To carry out the inference, Bayesian approach needs to start with a prior, whereas MaxEnt approach needs to start with a set of valid distributions. In practice people often use constant function for prior, and "distributions for which expected feature counts match observed feature counts" for the valid set of distributions.

Note that when dealing with continuous distributions, result of MaxEnt inference depends on the choice of measure over our set of distributions, so I think it's hard to justify in continuous case.

Here are some papers I scanned a while back on foundations of Bayesian and MaxEnt approaches.

24 comments:

Yaroslav Bulatov said...

Just came across this related post by Anand, apparently Feller was critical of Bayesian approach

http://ergodicity.net/2010/08/17/fellers-anti-bayesianism/

remo said...

THANKS FOR THE INFORMATION...
<a href="http://www.chloros.in/digital-marketing-internship.htmlhttp://www.chloros.in/digital-marketing-internship.html>Digital Marketing Internship Program in Bangalore</a>

draj said...



Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

Best Data Science Course Training in Hyderabad-360DigiTMG said...

I have recently started to read this blog, the info you provide about Machine Learning on this post has helped me greatly. Thanks for all of your time & work.Learn best
Data Science Course in Hyderabad using python and R programming

Best Business Analytics Course in Hyderabad said...

I feel really happy to have seen your post and look forward to so many more entertaining times reading here. Thanks once more for all the details.Learn best Business Analytics Course in Hyderabad

innomatics research labs said...


Data Science Course Training in Hyderabad


Data Science Training in Hyderabad, Data Science Course in Hyderabad, Data Science Training, Data Science Course, Data Science Institute, Data Science Certification




Data Science Course in Hyderabad

Data Science Training in Hyderabad
Data Science Course Training in Hyderabad


Advanced-Data science training with Free Internship & 100% Placement Assistance

About the Data Science course Training in Hyderabad

Data is everywhere, which is growing exponentially globally, and this may still grow at an accelerating rate for the foreseeable future. Businesses generate massive amounts of data within the type of blogs, messages, transaction documents, mobile device data, social media, etc. By using this data effectively, a firm can create vital value and grow their economy by enhancing productivity, increasing efficiency, and delivering more value to consumers.

Data Science helps in combining the disruption into categories and communicating their potential, which allows data and analytics leaders to drive better results. Top businesses thought there is a necessity to research the data for significant benefits. They use the insights from data for the advantage of users.

Human deciding is becoming increasingly inadequate to pander to a never-ending expansion of the data . However, Data Science and Machine Learning are excelling in solving highly complex data-rich problems. To think beyond the human brain and maintain the balance with the knowledge that's evolved, disrupted, and being employed the sectors altogether, data scientists foster new methodologies. Data scientists must try 'big data expeditions' to explore the data for previously undiscovered value - the first common application of data science. Typical applications include marketing segmentation, advertising, tweaking dynamic pricing models, or banks finding risks and adjusting the financial risk models.

What are the Tools utilized in Data Science?

360digitmg said...

I think about it is most required for making more on this get engaged
Best Data Science Courses in Hyderabad

traininginstitute said...

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
digital marketing courses in hyderabad with placement

Anonymous said...

Nice blog post,
Digital Marketing Trainer in Hyderabad

traininginstitute said...

I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place
digital marketing courses in hyderabad with placement


Priya Rathod said...

An amazing web journal I visit this blog, it's unbelievably wonderful.
DevOps Training in Hyderabad
DevOps Course in Hyderabad

Priya Rathod said...

Reading this article was very useful-- particularly for me, as it gave me some more insight into writing tips. I wanted to add that the author did a good job of explaining those points clearly. This is particularly helpful for new entrepreneurs because it shows how important it is to consider how everyone perceives your business from the outside. I'm looking forward reading other articles soon.
AWS Training in Hyderabad
AWS Course in Hyderabad

Ramesh Sampangi said...

Fantastic blog. Informative and Knowledgeable content. Keep sharing more blogs with us.
Machine Learning Course in Hyderabad

technologyforall said...
This comment has been removed by the author.
Ramesh Sampangi said...

Become a data science expert by joining AI Patasala’s Data Science Course Training in Hyderabad program, where data science aspirants can learn data science concepts with practical knowledge.
Data Science Course Training Institute in Hyderabad with Placements
Data Science Course Training in Hyderabad

faaantas said...




Hey friend, it is very well written article, thank you for the valuable and useful information you provide in this post. Keep up the good work! FYI, please check these depression, stress and anxiety related articles.
Federal Bank Signet Credit Card 2021 Review , The High Five Habit Free pdf Download , 10 lines about online classes in English

Unknown said...

I feel extremely appreciative that I read this. It is extremely useful and exceptionally enlightening and I truly took in a ton from it. data science course in surat

Unknown said...

I am truly getting a charge out of perusing your elegantly composed articles. It would seem that you burn through a ton of energy and time on your blog. I have bookmarked it and I am anticipating perusing new articles. Keep doing awesome. data science course in mysore

Project Management Course said...

You finished certain solid focuses there. I did a pursuit regarding the matter and discovered almost all people will concur with your blog.

traininginstitute said...

Nice knowledge gaining article. This post is really the best on this valuable topic.
data science training in malaysia

PMP Training in Malaysia said...

360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

Unknown said...

it was a wonderful chance to visit this kind of site and I am happy to know. thank you so much for giving us a chance to have this opportunity.. data science course in mysore

Mahil mithu said...

Well, I really appreciated for your great work. This topic submitted by you is helpful and keep sharing...
Separation Before Divorce
Cost of Legal Separation VS Divorce

Nigersaurus teeth said...

discussing the features and code of the new version of DBZ Adventures Unleashed Trello. The DBZ is a manga-based video game. The game is produced by Bandai Namco Entertainment etc. DBZ is an amine TV series that was launched by Japan.