## Saturday, April 29, 2006

### Naive Bayes vs. Logistic Regression

There's often confusion as to the nature of the differences between Logistic Regression and Naive Bayes Classifier. One way to look at it is that Logistic Regression and NBC consider the same hypothesis space, but use different loss functions, which leads to different models for some datasets.

To see that both logistic regression and naive bayes classifier consider the same hypothesis space we can rewrite Naive Bayes density as follows (restricting attention to binary domain):

where

Alternatively you can get exponential family parametrization from the observation that Naive Bayes model has no unshielded colliders, so undirected model obtained by dropping the arrows is equivalent on the set of positive densities.

From here you can rewrite it as product of conditional and marginal distributions

You can see that the conditional term is equivalent to logistic regression, so every possible conditional density that can be modelled by logistic regression can be modelled by Naive Bayes by setting \phi's aribrarily.

So both logistic regression and Naive Bayes have the same hypothesis space, but optimize different objective functions. In particular logistic regression maximizes

whereas Naive Bayes maximizies

You can see that the second term involves both phis and thetas, so conditional and marginal likelihoods are coupled. If empirical density is realizable by our model, then this coupling doesn't matter -- both conditional and marginal terms can achieve their respective maxima so Naive Bayes and Logistic Regression will produce the same estimate. However that not necessarily true when empirical density is unrealizable -- you may have to compromise between achieving high conditional likelihood and high marginal likelihood.

To get a generatively trained model that is identical to conditionally trained Naive Bayes, Minka suggests treating parameters in conditional and joint as separate sets.

So now we can introduce n new parameters so that conditional and marginal densities become decoupled.

It's interesting to see what is the new marginal density of our model. The marginal density marginalizes out Y. Since Y is the only separator in original graph, the resulting graph will be fully connected.

So if we were to restrict attention to linear exponential families, we'd have to introduce 2^n features. I'm wondering if it's possible to infer that fact by looking at the form of the density. You'd have to show that the size of sufficient statistic is at least 2^n, or alternatively show that the smallest linear space that embeds the unnormalized log densities has dimension 2^n

If x_i are real or complex valued, one could show that the smallest linear space that embeds the set above has to be infinite dimensional, by noting that since log(1+exp(ax)) has singularities at i pi a, so one can always construct linearly independent elements by choosing a accordingly. I wonder, are there similar tricks that would work for discrete x?

(derivations)

Anonymous said...

Hi Yaroslav,

I'm very interested in the post you published under the link below.

http://yaroslavvb.blogspot.com/2006/04/naive-bayes-vs-logistic-regression.html

(naive bayes vs. regression)

However, the link doesn't show the formulas.
Can you please send me the version with the formulas?

Shira
shirrra@gmail.com

gagan said...

amazing. Thanks.

Really helping with my ML course.

daryoush said...

When you say:

"...both logistic regression and naive bayes classifier consider the same hypothesis space"

I am not sure I understand the significance of "same hypothesis space" comment. When (or which algorithms) would have different hypothesis space.

Thanks,

Daryoush

Unknown said...

Actually there has some similarities between these two but none of them are not that bad than others in quality maters. more that I love for the helpful information and its really amazing.

Charles Kee said...

The post admission of jamb is here now and there has form for the admission and you should fill it up before reading the rules. https://pathology.residencypersonalstatements.net/pathology-residency-personal-statement-sample/ that is the best one place to know about the quality writing rules.

Anonymous said...

It is one of the easiest things to do the cover letter as the electronics sector is not that tough to functionized here. website go here and you'll be get and helpful ideas on the subject of academic papers writing.

Unknown said...

I think things like this are really interesting. I absolutely love to find unique places like this. It really looks super creepy though!!

Best Machine Learning institute in Chennai | machine learning with python course in chennai | best training institute for machine learning

smak said...

We can understand the formulas with the help of https://www.lorservice.com/our-recommendation-letter-service/letter-of-recommendation-for-nursing-school-writing-service/ because these formulas are for professionals and the professionals know how to use those and where to use those.

mikon said...

There is not just confusion but the difference is not clear that creates confusion but now https://www.nursingpaper.com/our-services/nursing-reflective-journal/ is there to clear such confusion and that can help the teachers to learn what is required in reality.

draj said...

Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

john said...
Mike Smith said...

Our Webroot Geek Squad Agents provide repair, installation and setup services on all kinds of tech at more than 1,100 Best Buy stores – including computer & tablet repair, setup and support, TV & home theater repair, car stereo & GPS installation, cell phone repair and home appliance repair. We fix most makes and models, no matter where you bought them and can show you how to get the most out of your technology. Want to fix any gadget issue? Our Trend Micro Geek Squad team is here to provide you with seamless services for your faulty devices. Geek Squad has set the benchmark when it comes to office or home appliances repairing. Our experts get acquainted with some exclusive technologies for hardware or electronic device issues. We can fix all the hindrances that can’t be solved by a typical customer support approach. The specialized touch of our technicians resolves the problem immediately.

beryltryphosa said...
brayne said...
Essay help said...

Your blogs are authentic. I love them .Are you also searching for essay help? we are the best solution for you. We are best known for delivering quality essay help.

dissertation writing help said...

I love it here. Keep sharing your good vibes. I love them Are you also searching for dissertation writing help? we are the best solution for you. We are best known for delivering cheap assignments to students without having to break the bank

help with my assignment said...

Your blogs are amazing. Keep sharing. I love them Are you also searching for help with my assignment? we are the best solution for you. We are best known for delivering the best urgent assignment help.

Nursing Writing Services said...

Your blogs are great.Are you also searching for Nursing Writing Services? we are the best solution for you. We are best known for delivering nursing writing services to students without having to break the bank.

helpmehempinc said...
helpmehempinc said...

WHITE GELATO PLUS
Dubz Garden and Gooniez Strain
OREOZ BY DUBZ GARDEN
Dubz Garden

counterfeit money
Australia Dollar | Fake Money AUD

DIRECT CONTCT
EMAIL........JONESHOBBIT55@GMAIL.COM
WICKR ID.....ALPHAPLUG007
TELEGRAM.....ALPHACASH0

Sultan said...

This is really a nice information on how to buy-bulk-cbn-isolate-powder in UK. Must necessary to share this blog, This comment will help to get full knowledge on other CBD EXTRACTS LIKE buy-bulk-cbg-isolate-crystals in UK,
buy-bulk-cbd-isolate-crystals UK, Pure THCA Diamonds in UK and Buy Bulk THCA Isolate | Powder | Online, will also help for the coming future THANKS.

CALL/TEXT/WHATSAPP >>>>>>>>>
+1(424) 235 3914

EMAIL >>>>>>>>> support@420labextracts.com

Elgin Bedou said...

Hello thanks for shearing this very important infor we are so pleased with your article Buy Bulk THCA Isolate | buy thca online | buy thca online uk | buy thca diamonds online, regarding thca isolate and we wish to inform you that we have interesting content regarding thca isolate powder and it might interest your audience who which to know more about the potency of the above mentioned productThca isolate powder.interested persons should
order thca isolate powder online, kindly reach our to us via the following listed contact details

Call/text/whatsapp....+1(424) 235 3914
email....support@420labextracts.com

Sultan said...

Hi there,
live resin for sale

delta 9 thc for sale

delta 8 thc for sale

CBN Isolate-Powder for sale

CBD isolates for sale

thca diamond for sale

thca isolate powder for sale

CALL/TEXT/WHATSAPP >>>>>>>>>
+1(424) 235 3914

EMAIL >>>>>>>>> support@420labextracts.com

phillipsgrey said...

I need to to thank you for this great read!! I definitely loved every bit of it.
EVERYTHING YOU NEED TO KNOW ABOUT BLUE COOKIES STRAIN I have got you bookmarked to look at new stuff you

smith said...

Hi there,
Thanks for shearing this very important information about CBG ISOLATE | CRYSTALS, i have been using this isolates for decates now and have some useful informations i will like to shear with you... read more...

jones said...

thanks for the wonderful article regarding thca isolate ,thca isolate ukwe wiush to let you know that we ,have great content which might interest you and your readers at legal420isolate ,thca near mefor more details contact us\

email...support@legal420isolate.com

call/text.....+1(505)257-5355

jones said...

thanks for your useful content about thca powderBuy THCA Isolate Powder online,wish to inform you that we have interesting article which can help add meaningful contribution

call/text....+1(505)257-5355

waxmann said...

Hello

Have some useful information i will like to shear with you about thca isolate read more.

For more details...

cotact: +1 (424) 235-3914

EMAIL: support@420labextracts.com

website: 420labextracts.com

waxmann said...

Hello

Have some useful information i will like to shear with you about buy bulk thca isolate | powder read more.

For more details...

cotact: +1 (424) 235-3914

EMAIL: support@420labextracts.com

website: 420labextracts.com

waxmann said...

Thanks for shearing such a useful content about pure thca diamond myself being a user of thca isolate for decades now, have got some useful articles and contents i will like to update you with

jones said...

Thanks for the useful information ,about Thca isolate, we have great article which can provide them with good informations about Thca isolate,Thca isolate powderfor more details .

DAVE said...

Thanks for shearing!!! its really helped people who dont know much about THCA ISOLATE. I will like to shear some our contents and articles about pure thca diamond with you

website: 420labextracts.com
contact: +1 (424) 235-3914

DAVE said...

Very interesting! this blog is very helpful for someone like me who have been doing researchwork on THCA ISOLATE powder.
We have some important articles and contents about pure thca diamond. read more...

website: 420labextracts.com
contact: +1 (424) 235-3914

Elgin Bedou said...

jones said...

Thanks for the wonderful article regarding , delta 9 thc cartridge,we have good content ,which might interest your audience , read more.

jones said...

jones said...

jones said...

Thanks for the interesting details ,Thca isolate powder,we have great and useful content which might , find more,

POT CITY HERBS said...

CBD oil can describe as a product derived from Cannabis. It’s a cannabinoid; the chemical compounds naturally present inside marijuana seeds. Although it’s derived from marijuana, CBD does not cause a “high” effect or any intoxication caused by another cannabinoid called THC . IMPORTANCE & BENEFITS OF CBD OIL CBD might be able to assist you in managing anxiety. Researchers think that reliable Sources could alter how your brain’s receptors react to serotonin, a substance associated with mental well-being. Receptors are tiny proteins that attach to cells.

jones said...

thanks for the wonderful article regarding ,delta 9 for sale, we have interesting content which might provide more insight regarding the above mentioned product, find more,

Call/ text >>>>>>+1(505)257-5355

Email >>>>>support@legal420isolate.com

jones said...

Thanks for the wonderful informations regarding ,order thca isolate powder,we intereting content which can donate meaningful contribution regarding the above mentioned , see more,

Email>>>support@legal420isolate.com

call/text>>>>1(505)257-5355