Foundations of probabilistic inference is often a subject of much disagreement, with some leading Bayesian sometimes going as far as to say that MaxEnt method doesn't make any sense, and the MaxEnt camp picking on the issue of subjectivity.

The way I see it, MaxEnt and Bayesian approaches are just different ways of using external information to pick a probability distribution.

With Bayesian approach, you are given data and a prior. Bayesian inference then comes out as a natural extension of logic to probabilities. See Ch.1 of Cox's Algebra of Probable Inference for full axiomatic derivation.

With MaxEnt approach, you are given a set of valid probability distributions. You can then justify MaxEnt axiomatically but personally I like the motivation given in Jaynes book, Probability Theory: The Logic of Science (in particular Ch.11).

Roughly speaking, it goes as follows. Suppose we have a distibution $p$ over $d$ outcomes. We say that a sequence of $n$ observations is explained by distribution $p$ if relative counts of outcomes in this sequence match $p$ exactly. The number of sequences matched by $p$ is the following quantity

$$c(p)=\frac{n!}{(np_1)!(np_2)!\cdots (np_d)!}$$

If we have a set of valid distributions $p$, it then makes sense to pick the one with highest $c(p)$ because it'll fit the greatest number of possible future datasets. Approximating $n!$ by $n^n$ we get the following

$$\frac{1}{n}\log c(p)\approx \frac{1}{n}\log \frac{n^n}{(np_1)^{np_1}(np_2)^{np_2}\cdots (np_d)^{np_d}}=H(p)$$

Hence picking distribution with highest H(p) usually means picking the distribution that will be able to explain the most possible observation sequences. Because it's an approximation, there can be exceptions, for instance distributions $(\frac{12}{18},\frac{3}{18},\frac{3}{18})$ and $(\frac{9}{18},\frac{8}{18},\frac{1}{18})$ have equal entropy, but the latter explains 18% more length 18 sequences.

To carry out the inference, Bayesian approach needs to start with a prior, whereas MaxEnt approach needs to start with a set of valid distributions. In practice people often use constant function for prior, and "distributions for which expected feature counts match observed feature counts" for the valid set of distributions.

Note that when dealing with continuous distributions, result of MaxEnt inference depends on the choice of measure over our set of distributions, so I think it's hard to justify in continuous case.

Here are some papers I scanned a while back on foundations of Bayesian and MaxEnt approaches.

## 43 comments:

Just came across this related post by Anand, apparently Feller was critical of Bayesian approach

http://ergodicity.net/2010/08/17/fellers-anti-bayesianism/

THANKS FOR THE INFORMATION...

<a href="http://www.chloros.in/digital-marketing-internship.htmlhttp://www.chloros.in/digital-marketing-internship.html>Digital Marketing Internship Program in Bangalore</a>

Excellent machine learning blog,thanks for sharing...

Seo Internship in Bangalore

Smo Internship in Bangalore

Digital Marketing Internship Program in Bangalore

I have recently started to read this blog, the info you provide about Machine Learning on this post has helped me greatly. Thanks for all of your time & work.Learn best

Data Science Course in Hyderabad using python and R programming

I feel really happy to have seen your post and look forward to so many more entertaining times reading here. Thanks once more for all the details.Learn best Business Analytics Course in Hyderabad

Data Science Course Training in Hyderabad

Data Science Training in Hyderabad, Data Science Course in Hyderabad, Data Science Training, Data Science Course, Data Science Institute, Data Science Certification

Data Science Course in Hyderabad

Data Science Training in Hyderabad

Data Science Course Training in Hyderabad

Advanced-Data science training with Free Internship & 100% Placement Assistance

About the Data Science course Training in Hyderabad

Data is everywhere, which is growing exponentially globally, and this may still grow at an accelerating rate for the foreseeable future. Businesses generate massive amounts of data within the type of blogs, messages, transaction documents, mobile device data, social media, etc. By using this data effectively, a firm can create vital value and grow their economy by enhancing productivity, increasing efficiency, and delivering more value to consumers.

Data Science helps in combining the disruption into categories and communicating their potential, which allows data and analytics leaders to drive better results. Top businesses thought there is a necessity to research the data for significant benefits. They use the insights from data for the advantage of users.

Human deciding is becoming increasingly inadequate to pander to a never-ending expansion of the data . However, Data Science and Machine Learning are excelling in solving highly complex data-rich problems. To think beyond the human brain and maintain the balance with the knowledge that's evolved, disrupted, and being employed the sectors altogether, data scientists foster new methodologies. Data scientists must try 'big data expeditions' to explore the data for previously undiscovered value - the first common application of data science. Typical applications include marketing segmentation, advertising, tweaking dynamic pricing models, or banks finding risks and adjusting the financial risk models.

What are the Tools utilized in Data Science?

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science institutes in Hyderabad

I think about it is most required for making more on this get engaged

Best Data Science Courses in Hyderabad

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science courses in Hyderabad

All things considered I read it yesterday however I had a few considerations about it and today I needed to peruse it again on the grounds that it is very elegantly composed.

data scientist training

Thanks for posting the best information and the blog is very informative.

Data science course in FaridabadActually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.

digital marketing courses in hyderabad with placement

Thanks for posting the best information and the blog is very important.

data science institutes in hyderabadInformative blog

data analytics courses in hyderabad

Nice blog post,

Digital Marketing Trainer in Hyderabad

I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place

digital marketing courses in hyderabad with placement

An amazing web journal I visit this blog, it's unbelievably wonderful.

DevOps Training in Hyderabad

DevOps Course in Hyderabad

Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

data science course in varanasi

Fantastic article I ought to say and thanks to the info. Instruction is absolutely a sticky topic. But remains one of the top issues of the time. I love your article and look forward to more.

Data Science Course in Bangalore

Thank you quite much for discussing this type of helpful informative article. Will certainly stored and reevaluate your Website.

Data Analytics Course in Bangalore

Very wonderful informative article. I appreciated looking at your article. Very wonderful reveal. I would like to twit this on my followers. Many thanks! .

Data Analytics training in Bangalore

Excellent blog and article. I enjoy looking at your posts. Thanks for sharing on the web. I am hoping the same best work from you in the future as well. Thank you for sharing great blogging with us!

Data Science Training in Hyderabad

Data Science Course in Hyderabad

Reading this article was very useful-- particularly for me, as it gave me some more insight into writing tips. I wanted to add that the author did a good job of explaining those points clearly. This is particularly helpful for new entrepreneurs because it shows how important it is to consider how everyone perceives your business from the outside. I'm looking forward reading other articles soon.

AWS Training in Hyderabad

AWS Course in Hyderabad

Excellent post.I want to thank you for this informative read, I really appreciate sharing this great post.Keep up your work

data scientist course in hyderabad

Thanks for posting the best information and the blog is very good.

data science course in LucknowNice to be visiting your blog once more, it has been months for me. Well this article that ive been waited for therefore long. i want this article to finish my assignment within the faculty, and it has same topic together with your article. Thanks, nice share.

data scientist course in hyderabad

Fantastic blog. Informative and Knowledgeable content. Keep sharing more blogs with us.

Machine Learning Course in Hyderabad

Become a data science expert by joining AI Patasala’s Data Science Course Training in Hyderabad program, where data science aspirants can learn data science concepts with practical knowledge.

Data Science Course Training Institute in Hyderabad with Placements

Data Science Course Training in Hyderabad

Hey friend, it is very well written article, thank you for the valuable and useful information you provide in this post. Keep up the good work! FYI, please check these depression, stress and anxiety related articles.

Federal Bank Signet Credit Card 2021 Review , The High Five Habit Free pdf Download , 10 lines about online classes in English

Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.business analytics course in rohtak

I feel extremely appreciative that I read this. It is extremely useful and exceptionally enlightening and I truly took in a ton from it. data science course in surat

I am truly getting a charge out of perusing your elegantly composed articles. It would seem that you burn through a ton of energy and time on your blog. I have bookmarked it and I am anticipating perusing new articles. Keep doing awesome. data science course in mysore

You finished certain solid focuses there. I did a pursuit regarding the matter and discovered almost all people will concur with your blog.

Nice knowledge gaining article. This post is really the best on this valuable topic.

data science training in malaysia

360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

All things considered I read it yesterday yet I had a few musings about it and today I needed to peruse it again in light of the fact that it is very elegantly composed. https://360digitmg.com/course/data-analytics-using-python-r

I truly appreciate just perusing the entirety of your weblogs. Just needed to educate you that you have individuals like me who value your work. Unquestionably an extraordinary post. Caps off to you! The data that you have given is exceptionally useful.data science course in bhopal

Excellent effort to make this blog more wonderful and attractive.

full stack developer course with placement

it was a wonderful chance to visit this kind of site and I am happy to know. thank you so much for giving us a chance to have this opportunity.. data science course in mysore

Very informative message! There is so much information here that can help any business start a successful social media campaign!

data science training in london

I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… PMP certification in malaysia

Well, I really appreciated for your great work. This topic submitted by you is helpful and keep sharing...

Separation Before Divorce

Cost of Legal Separation VS Divorce

Post a Comment