Saturday, April 29, 2006

Naive Bayes vs. Logistic Regression

There's often confusion as to the nature of the differences between Logistic Regression and Naive Bayes Classifier. One way to look at it is that Logistic Regression and NBC consider the same hypothesis space, but use different loss functions, which leads to different models for some datasets.




To see that both logistic regression and naive bayes classifier consider the same hypothesis space we can rewrite Naive Bayes density as follows (restricting attention to binary domain):



where



Alternatively you can get exponential family parametrization from the observation that Naive Bayes model has no unshielded colliders, so undirected model obtained by dropping the arrows is equivalent on the set of positive densities.

From here you can rewrite it as product of conditional and marginal distributions




You can see that the conditional term is equivalent to logistic regression, so every possible conditional density that can be modelled by logistic regression can be modelled by Naive Bayes by setting \phi's aribrarily.


So both logistic regression and Naive Bayes have the same hypothesis space, but optimize different objective functions. In particular logistic regression maximizes




whereas Naive Bayes maximizies




You can see that the second term involves both phis and thetas, so conditional and marginal likelihoods are coupled. If empirical density is realizable by our model, then this coupling doesn't matter -- both conditional and marginal terms can achieve their respective maxima so Naive Bayes and Logistic Regression will produce the same estimate. However that not necessarily true when empirical density is unrealizable -- you may have to compromise between achieving high conditional likelihood and high marginal likelihood.

To get a generatively trained model that is identical to conditionally trained Naive Bayes, Minka suggests treating parameters in conditional and joint as separate sets.

So now we can introduce n new parameters so that conditional and marginal densities become decoupled.



It's interesting to see what is the new marginal density of our model. The marginal density marginalizes out Y. Since Y is the only separator in original graph, the resulting graph will be fully connected.



So if we were to restrict attention to linear exponential families, we'd have to introduce 2^n features. I'm wondering if it's possible to infer that fact by looking at the form of the density. You'd have to show that the size of sufficient statistic is at least 2^n, or alternatively show that the smallest linear space that embeds the unnormalized log densities has dimension 2^n



If x_i are real or complex valued, one could show that the smallest linear space that embeds the set above has to be infinite dimensional, by noting that since log(1+exp(ax)) has singularities at i pi a, so one can always construct linearly independent elements by choosing a accordingly. I wonder, are there similar tricks that would work for discrete x?

(derivations)

41 comments:

Anonymous said...

Hi Yaroslav,

I'm very interested in the post you published under the link below.

http://yaroslavvb.blogspot.com/2006/04/naive-bayes-vs-logistic-regression.html

(naive bayes vs. regression)

However, the link doesn't show the formulas.
Can you please send me the version with the formulas?

thanks in advance,

Shira
shirrra@gmail.com

gagan said...

amazing. Thanks.

Really helping with my ML course.

daryoush said...

When you say:

"...both logistic regression and naive bayes classifier consider the same hypothesis space"

I am not sure I understand the significance of "same hypothesis space" comment. When (or which algorithms) would have different hypothesis space.

Thanks,

Daryoush

Unknown said...

Actually there has some similarities between these two but none of them are not that bad than others in quality maters. more that I love for the helpful information and its really amazing.

Charles Kee said...

The post admission of jamb is here now and there has form for the admission and you should fill it up before reading the rules. https://pathology.residencypersonalstatements.net/pathology-residency-personal-statement-sample/ that is the best one place to know about the quality writing rules.

Anonymous said...

It is one of the easiest things to do the cover letter as the electronics sector is not that tough to functionized here. website go here and you'll be get and helpful ideas on the subject of academic papers writing.

Unknown said...

I think things like this are really interesting. I absolutely love to find unique places like this. It really looks super creepy though!!


Best Machine Learning institute in Chennai | machine learning with python course in chennai | best training institute for machine learning

smak said...

We can understand the formulas with the help of https://www.lorservice.com/our-recommendation-letter-service/letter-of-recommendation-for-nursing-school-writing-service/ because these formulas are for professionals and the professionals know how to use those and where to use those.

mikon said...

There is not just confusion but the difference is not clear that creates confusion but now https://www.nursingpaper.com/our-services/nursing-reflective-journal/ is there to clear such confusion and that can help the teachers to learn what is required in reality.

draj said...



Excellent machine learning blog,thanks for sharing...
Seo Internship in Bangalore
Smo Internship in Bangalore
Digital Marketing Internship Program in Bangalore

john said...


Great Article
IEEE final year projects on machine learning


JavaScript Training in Chennai

Final Year Project Centers in Chennai



JavaScript Training in Chennai

Mike Smith said...

Our Webroot Geek Squad Agents provide repair, installation and setup services on all kinds of tech at more than 1,100 Best Buy stores – including computer & tablet repair, setup and support, TV & home theater repair, car stereo & GPS installation, cell phone repair and home appliance repair. We fix most makes and models, no matter where you bought them and can show you how to get the most out of your technology. Want to fix any gadget issue? Our Trend Micro Geek Squad team is here to provide you with seamless services for your faulty devices. Geek Squad has set the benchmark when it comes to office or home appliances repairing. Our experts get acquainted with some exclusive technologies for hardware or electronic device issues. We can fix all the hindrances that can’t be solved by a typical customer support approach. The specialized touch of our technicians resolves the problem immediately.

beryltryphosa said...

snowflake interview questions
riya sold her car
zensoft interview questions

brayne said...

buy weed online


buy real weed online


buy synthetic weed online


how to buy weed online


order marijuana online


buy cbd oil online


order cbd oil online


real marijuana online


different types of weed


weed for sale online


buy edibles online


buying weed online reviews


order marijuana online


real marijuana online


buy marijuana online


cannabis oil for sale


order wax and shatter


marijuana for sale


purple og kush


cbd oil for sale


marijuana edibles for sale


buy legal buds online


buy medical marijuana online


buy medical weed online


where to buy marijuana online


buy wax and shatter


order cannabis cbd


wax and shatter for sale


buy real weed online


og kush breath


buy medical marijuana


buy weed edibles online


buy cannabis online


can you buy weed online


buying weed online


order edibles online


order marijuana online without a medical card


how to order marijuana online


can you buy marijuana online


purchase cannabis oil


cannabis edibles for sale


og kush seeds


OG Kush Effects

Essay help said...

Your blogs are authentic. I love them .Are you also searching for essay help? we are the best solution for you. We are best known for delivering quality essay help.

dissertation writing help said...

I love it here. Keep sharing your good vibes. I love them Are you also searching for dissertation writing help? we are the best solution for you. We are best known for delivering cheap assignments to students without having to break the bank

help with my assignment said...

Your blogs are amazing. Keep sharing. I love them Are you also searching for help with my assignment? we are the best solution for you. We are best known for delivering the best urgent assignment help.

Nursing Writing Services said...

Your blogs are great.Are you also searching for Nursing Writing Services? we are the best solution for you. We are best known for delivering nursing writing services to students without having to break the bank.

helpmehempinc said...

WHERE CAN I BUY ROXICODONE ONLINE WITHOUT PRESCRIPTION
Buy 5F-MDEMB-2201
Buy real Passport near me
WHITE GELATO PLUS

helpmehempinc said...


WHITE GELATO PLUS
Dubz Garden and Gooniez Strain
OREOZ BY DUBZ GARDEN
Dubz Garden

Buy 5F-MDEMB-2201
buy 4FAKB
Buy JWH-018

counterfeit money
Canada Dollar | Fake Money CAD
Australia Dollar | Fake Money AUD
Buy real Passport near me


DIRECT CONTCT
EMAIL........JONESHOBBIT55@GMAIL.COM
WICKR ID.....ALPHAPLUG007
TELEGRAM.....ALPHACASH0

Sultan said...

This is really a nice information on how to buy-bulk-cbn-isolate-powder in UK. Must necessary to share this blog, This comment will help to get full knowledge on other CBD EXTRACTS LIKE buy-bulk-cbg-isolate-crystals in UK,
buy-bulk-cbd-isolate-crystals UK, Pure THCA Diamonds in UK and Buy Bulk THCA Isolate | Powder | Online, will also help for the coming future THANKS.

For more infor please contact the following.....

CALL/TEXT/WHATSAPP >>>>>>>>>
+1(424) 235 3914

EMAIL >>>>>>>>> support@420labextracts.com

Elgin Bedou said...

Hello thanks for shearing this very important infor we are so pleased with your article Buy Bulk THCA Isolate | buy thca online | buy thca online uk | buy thca diamonds online, regarding thca isolate and we wish to inform you that we have interesting content regarding thca isolate powder and it might interest your audience who which to know more about the potency of the above mentioned productThca isolate powder.interested persons should
order thca isolate powder online, kindly reach our to us via the following listed contact details

Call/text/whatsapp....+1(424) 235 3914
email....support@420labextracts.com

Sultan said...

Hi there,
Thank you so much for the post you do and also I like your post, Are you looking for bulk cbc isolate, marijuanna seedbuy bulk cbg, thca powder for sale, cbg bulk,laboratory products online, buy lab extarcts online, buy, bulk Live Resin online, buy quality delta 8 online, buy Bulk Delta 9 THC online, buy delta-9 cartridge legal online, buygoldie.locks.extracts online, buy honey creek labs product online, lab supplies online, buy, bulk resin online, buy pharma grade products online, pharma grade supplements, buy pharma grade supplements online, pharmaceutical grade supplements canada, buy bulk CBN Isolate Powder, buy onlinecbn isolate wholesale, buy bulk CBC Isolate | Powder, cbd isolate wholesale online prices, buy cbd isolate crystal online, buy cbd isolate tincture online, buy cbd distillates online, buy THCD jars online, buy bulk Delta 8 THC online, buy Bulk Delta 9 online, buy bulk delta 9 THC online, buy Bulk CBD Jars online, buy Bulk THCA Powder online, buy THCA crystallineonline, where to buy thca diamonds, thca diamonds near me, buy Bulk CBD Isolate online, buy CBD powder online, best Places CBD Powder online, best cbd isolate, buy best cbd isolate online, where can i buy thca oil, buy cbn isolate dosage online forsale, buy cbn distillate online forsale, isolate crystals, thca isolate wholesale, goldy locks cbg, kannabia wholesale, wholesale cbc isolate, cbc wholesale, wholesale cbc isolate,thca powder for sale with the well price and our services are very fast.
Click here for..
live resin for sale


delta 9 thc for sale


delta 8 thc for sale


CBN Isolate-Powder for sale


CBD isolates for sale



thca diamond for sale


thca isolate powder for sale

For more infor please contact the following.....

CALL/TEXT/WHATSAPP >>>>>>>>>
+1(424) 235 3914

EMAIL >>>>>>>>> support@420labextracts.com

phillipsgrey said...

I need to to thank you for this great read!! I definitely loved every bit of it.
EVERYTHING YOU NEED TO KNOW ABOUT BLUE COOKIES STRAIN I have got you bookmarked to look at new stuff you

smith said...

Hi there,
Thanks for shearing this very important information about CBG ISOLATE | CRYSTALS, i have been using this isolates for decates now and have some useful informations i will like to shear with you... read more...

jones said...

thanks for the wonderful article regarding thca isolate ,thca isolate ukwe wiush to let you know that we ,have great content which might interest you and your readers at legal420isolate ,thca near mefor more details contact us\

email...support@legal420isolate.com

call/text.....+1(505)257-5355

jones said...



thanks for your useful content about thca powderBuy THCA Isolate Powder online,wish to inform you that we have interesting article which can help add meaningful contribution
for more details contact us via the following ..

call/text....+1(505)257-5355
email address...support@legal420isolate.com

waxmann said...

Hello

Have some useful information i will like to shear with you about thca isolate read more.


For more details...

cotact: +1 (424) 235-3914

EMAIL: support@420labextracts.com

website: 420labextracts.com

waxmann said...

Hello

Have some useful information i will like to shear with you about buy bulk thca isolate | powder read more.


For more details...

cotact: +1 (424) 235-3914

EMAIL: support@420labextracts.com

website: 420labextracts.com

waxmann said...

Thanks for shearing such a useful content about pure thca diamond myself being a user of thca isolate for decades now, have got some useful articles and contents i will like to update you with
buy thca isolate online...

jones said...

Thanks for the useful information ,about Thca isolate, we have great article which can provide them with good informations about Thca isolate,Thca isolate powderfor more details .

DAVE said...

Thanks for shearing!!! its really helped people who dont know much about THCA ISOLATE. I will like to shear some our contents and articles about pure thca diamond with you
read more...

website: 420labextracts.com
contact: +1 (424) 235-3914

DAVE said...

Very interesting! this blog is very helpful for someone like me who have been doing researchwork on THCA ISOLATE powder.
We have some important articles and contents about pure thca diamond. read more...


website: 420labextracts.com
contact: +1 (424) 235-3914

Elgin Bedou said...

Thanks for shearing this article about THCA ISOLATE, chech out some of our must important contentss about this product read more...

jones said...

Thanks for the wonderful article regarding , delta 9 thc cartridge,we have good content ,which might interest your audience , read more.

jones said...

Thanks for the wonderful informations about ,Thca isolate powder, We have quality content that will help you and your audience , read more,

jones said...

thanks for your interest in ,Thca isolate powder, we have in tersting content which can help you an your audience , read more,

jones said...

Thanks for the interesting details ,Thca isolate powder,we have great and useful content which might , find more,

POT CITY HERBS said...

CBD oil can describe as a product derived from Cannabis. It’s a cannabinoid; the chemical compounds naturally present inside marijuana seeds. Although it’s derived from marijuana, CBD does not cause a “high” effect or any intoxication caused by another cannabinoid called THC . IMPORTANCE & BENEFITS OF CBD OIL CBD might be able to assist you in managing anxiety. Researchers think that reliable Sources could alter how your brain’s receptors react to serotonin, a substance associated with mental well-being. Receptors are tiny proteins that attach to cells.

jones said...

thanks for the wonderful article regarding ,delta 9 for sale, we have interesting content which might provide more insight regarding the above mentioned product, find more,


Call/ text >>>>>>+1(505)257-5355

Email >>>>>support@legal420isolate.com

jones said...

Thanks for the wonderful informations regarding ,order thca isolate powder,we intereting content which can donate meaningful contribution regarding the above mentioned , see more,


Email>>>support@legal420isolate.com


call/text>>>>1(505)257-5355