Sunday, September 05, 2010

MaxEnt or Bayesian?

Foundations of probabilistic inference is often a subject of much disagreement, with some leading Bayesian sometimes going as far as to say that MaxEnt method doesn't make any sense, and the MaxEnt camp picking on the issue of subjectivity.

The way I see it, MaxEnt and Bayesian approaches are just different ways of using external information to pick a probability distribution.

With Bayesian approach, you are given data and a prior. Bayesian inference then comes out as a natural extension of logic to probabilities. See Ch.1 of Cox's Algebra of Probable Inference for full axiomatic derivation.

With MaxEnt approach, you are given a set of valid probability distributions. You can then justify MaxEnt axiomatically but personally I like the motivation given in Jaynes book, Probability Theory: The Logic of Science (in particular Ch.11).

Roughly speaking, it goes as follows. Suppose we have a distibution $p$ over $d$ outcomes. We say that a sequence of $n$ observations is explained by distribution $p$ if relative counts of outcomes in this sequence match $p$ exactly. The number of sequences matched by $p$ is the following quantity

$$c(p)=\frac{n!}{(np_1)!(np_2)!\cdots (np_d)!}$$

If we have a set of valid distributions $p$, it then makes sense to pick the one with highest $c(p)$ because it'll fit the greatest number of possible future datasets. Approximating $n!$ by $n^n$ we get the following

$$\frac{1}{n}\log c(p)\approx \frac{1}{n}\log \frac{n^n}{(np_1)^{np_1}(np_2)^{np_2}\cdots (np_d)^{np_d}}=H(p)$$

Hence picking distribution with highest H(p) usually means picking the distribution that will be able to explain the most possible observation sequences. Because it's an approximation, there can be exceptions, for instance distributions $(\frac{12}{18},\frac{3}{18},\frac{3}{18})$ and $(\frac{9}{18},\frac{8}{18},\frac{1}{18})$ have equal entropy, but the latter explains 18% more length 18 sequences.

To carry out the inference, Bayesian approach needs to start with a prior, whereas MaxEnt approach needs to start with a set of valid distributions. In practice people often use constant function for prior, and "distributions for which expected feature counts match observed feature counts" for the valid set of distributions.

Note that when dealing with continuous distributions, result of MaxEnt inference depends on the choice of measure over our set of distributions, so I think it's hard to justify in continuous case.

Here are some papers I scanned a while back on foundations of Bayesian and MaxEnt approaches.

29 comments:

Yaroslav Bulatov said...

Just came across this related post by Anand, apparently Feller was critical of Bayesian approach

http://ergodicity.net/2010/08/17/fellers-anti-bayesianism/

remo said...

THANKS FOR THE INFORMATION...
<a href="http://www.chloros.in/digital-marketing-internship.htmlhttp://www.chloros.in/digital-marketing-internship.html>Digital Marketing Internship Program in Bangalore</a>

draj said...



Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

Best Data Science Course Training in Hyderabad-360DigiTMG said...

I have recently started to read this blog, the info you provide about Machine Learning on this post has helped me greatly. Thanks for all of your time & work.Learn best
Data Science Course in Hyderabad using python and R programming

Best Business Analytics Course in Hyderabad said...

I feel really happy to have seen your post and look forward to so many more entertaining times reading here. Thanks once more for all the details.Learn best Business Analytics Course in Hyderabad

innomatics research labs said...


Data Science Course Training in Hyderabad


Data Science Training in Hyderabad, Data Science Course in Hyderabad, Data Science Training, Data Science Course, Data Science Institute, Data Science Certification




Data Science Course in Hyderabad

Data Science Training in Hyderabad
Data Science Course Training in Hyderabad


Advanced-Data science training with Free Internship & 100% Placement Assistance

About the Data Science course Training in Hyderabad

Data is everywhere, which is growing exponentially globally, and this may still grow at an accelerating rate for the foreseeable future. Businesses generate massive amounts of data within the type of blogs, messages, transaction documents, mobile device data, social media, etc. By using this data effectively, a firm can create vital value and grow their economy by enhancing productivity, increasing efficiency, and delivering more value to consumers.

Data Science helps in combining the disruption into categories and communicating their potential, which allows data and analytics leaders to drive better results. Top businesses thought there is a necessity to research the data for significant benefits. They use the insights from data for the advantage of users.

Human deciding is becoming increasingly inadequate to pander to a never-ending expansion of the data . However, Data Science and Machine Learning are excelling in solving highly complex data-rich problems. To think beyond the human brain and maintain the balance with the knowledge that's evolved, disrupted, and being employed the sectors altogether, data scientists foster new methodologies. Data scientists must try 'big data expeditions' to explore the data for previously undiscovered value - the first common application of data science. Typical applications include marketing segmentation, advertising, tweaking dynamic pricing models, or banks finding risks and adjusting the financial risk models.

What are the Tools utilized in Data Science?

prathyusha said...

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science institutes in Hyderabad

360digitmg said...

I think about it is most required for making more on this get engaged
Best Data Science Courses in Hyderabad

prathyusha said...

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science courses in Hyderabad

Maneesha said...

All things considered I read it yesterday however I had a few considerations about it and today I needed to peruse it again on the grounds that it is very elegantly composed.
data scientist training

Mallela said...

Thanks for posting the best information and the blog is very informative.Data science course in Faridabad

trainingcourses said...

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
digital marketing courses in hyderabad with placement

Mallela said...

Thanks for posting the best information and the blog is very important.data science institutes in hyderabad

Deekshitha said...

Informative blog
data analytics courses in hyderabad

Anonymous said...

Nice blog post,
Digital Marketing Trainer in Hyderabad

trainingcourses said...

I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place
digital marketing courses in hyderabad with placement


Priya Rathod said...

An amazing web journal I visit this blog, it's unbelievably wonderful.
DevOps Training in Hyderabad
DevOps Course in Hyderabad

Data Science Course in Bhilai - 360DigiTMG said...

Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

data science course in varanasi

DataScienceBOX said...

Fantastic article I ought to say and thanks to the info. Instruction is absolutely a sticky topic. But remains one of the top issues of the time. I love your article and look forward to more.
Data Science Course in Bangalore

DataScienceEXPO said...

Thank you quite much for discussing this type of helpful informative article. Will certainly stored and reevaluate your Website.
Data Analytics Course in Bangalore

DataBank said...

Nice to be seeing your site once again, it's been weeks for me. This article which ive been waited for so long. I need this guide to complete my mission inside the school, and it's same issue together along with your essay. Thanks, pleasant share.
Data Science training in Bangalore

DataCharge said...

Very wonderful informative article. I appreciated looking at your article. Very wonderful reveal. I would like to twit this on my followers. Many thanks! .
Data Analytics training in Bangalore

Priya Rathod said...

Excellent blog and article. I enjoy looking at your posts. Thanks for sharing on the web. I am hoping the same best work from you in the future as well. Thank you for sharing great blogging with us!
Data Science Training in Hyderabad
Data Science Course in Hyderabad

Priya Rathod said...

Reading this article was very useful-- particularly for me, as it gave me some more insight into writing tips. I wanted to add that the author did a good job of explaining those points clearly. This is particularly helpful for new entrepreneurs because it shows how important it is to consider how everyone perceives your business from the outside. I'm looking forward reading other articles soon.
AWS Training in Hyderabad
AWS Course in Hyderabad

data scientist course said...

Excellent post.I want to thank you for this informative read, I really appreciate sharing this great post.Keep up your work
data scientist course in hyderabad

Mallela said...

Thanks for posting the best information and the blog is very good.data science course in Lucknow

data scientist course said...

Nice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share.
data scientist course in hyderabad

Ramesh Sampangi said...

Fantastic blog. Informative and Knowledgeable content. Keep sharing more blogs with us.
Machine Learning Course in Hyderabad

technologyforall said...

Join our Online Data Science Course program to analyze data, make effective predictions and gain a better understanding of market trends. Create disruptive business models for the topmost industries as we equip you with a sound business analytical & entrepreneurial ideology, alongside an excellent grasp of marketing strategies and trends.