tag:blogger.com,1999:blog-105608002012-01-17T03:58:30.188-08:00Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.comBlogger1121100tag:blogger.com,1999:blog-10560800.post-63041105929988165222011-11-21T22:23:00.001-08:002011-11-21T22:29:29.146-08:00

There's a number of accepted papers whose camera-ready versions have been posted already. Here are the ones I found interesting. I'll give further update on these after the conference. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials, P. Krähenbühl, V. Koltun Fast and Accurate k-means For Large Datasets, M. Shindler, A. Wong, A. Meyerson Hashing Algorithms for

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-88703144894392474002011-11-13T22:39:00.001-08:002011-11-14T11:03:31.680-08:00

Here's a cool tool I stumbled across reading John Cook's blog -- Shape Catcher looks up Unicode value from a drawing of a character. Apparently it uses Shape Context features. This motivated me to put together another dataset, unlike notMNIST this focuses on the tail end of Unicode, this is 370k bitmaps representing 29k Unicode values, grouped by Unicode Unicode 370k

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-16032480614038453972011-11-09T13:17:00.000-08:002011-11-13T17:55:06.918-08:00

This is a dataset of scans of 1000 public domain books that was released to the public at ICDAR 2007. At the time there was no public serving infrastructure, so few people actually got the 120GB dataset. It has since been hosted on Google Cloud Storage and made available for public download http://commondatastorage.googleapis.com/books/icdar2007/README.txt http://

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-78651896744266854652011-11-06T23:01:00.000-08:002011-11-06T23:03:05.519-08:00

Below is an illustration of b-matching from (Huang,Jebara AISTATS 2007) paper. You start with a weighted graph and the goal is to connect each v to k u's to minimize total edge cost. If v's represent labelled datapoints, u's unlabeled and weights correspond to distances, this works as a robust version of kNN classifier (k=2 in the picture) because it prevents any datapoint from exhibiting too

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-19136715096112685212011-10-25T21:07:00.000-07:002011-10-26T11:43:54.839-07:00

My group has intern openings for winter and summer. Winter may be too late (but if you really want winter, ping me and I'll find out feasibility). We use OCR for Google Books, frames from YouTube videos, spam images, unreadable PDFs encountered by the crawler, images from Google's StreetView cameras, Android and few other areas. Recognizing individual character candidates is a key step in OCR

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-59717131349975495052011-09-24T12:25:00.000-07:002011-09-24T19:03:39.826-07:00

A discussion came up on Guido von Rossum's Google Plus post. It comes down to the fact that 2.1 is not exactly represented as a floating point number. Internally it's 2.0999999999999996, and this causes unexpected behavior. These kinds of issues often come up. The confusion is caused by treating floating point numbers as exact numbers, and expecting calculations with them to produce results

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-63806375229361894902011-09-08T22:45:00.000-07:002011-09-08T23:05:49.461-07:00

I've taken some publicly available fonts and extracted glyphs from them to make a dataset similar to MNIST. There are 10 classes, with letters A-J taken from different fonts. Here are some examples of letter "A" Judging by the examples, one would expect this to be a harder task than MNIST. This seems to be the case -- logistic regression on top of stacked auto-encoder with fine-tuning gets

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com9tag:blogger.com,1999:blog-10560800.post-62703744389117602632011-08-16T19:54:00.000-07:002011-08-17T00:24:58.342-07:00

In the old days you could statically link your program and run it on another Unix station without worrying about dependencies. Unfortunately static linking no longer works, so you need to make sure that your target platform has the right libraries. For instance, in order to get Matlab compiled code running on a server, you have to copy over libraries and set environment variables as specified

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-84381660305034175002011-07-13T01:52:00.000-07:002011-07-14T20:01:31.138-07:00

Google+ seems to have a fair number of Machine Learning people, I was able to track down 50 people I've met at conferences by starting at Andrew McCallum's circles. If you add me on Google Circles I'll assume you came from this blog and add you to my "Machine Learning" circle

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com5tag:blogger.com,1999:blog-10560800.post-76339896482667446202011-06-25T22:33:00.001-07:002011-07-14T20:16:58.126-07:00

Computers are supposed to be deterministic. This is often the case for single processor machines. However, as you scale up, guaranteeing determinism becomes increasingly expensive.Even on single processor machines you are facing non-determinism on semi-regular basis. Here are some examples Bugs + poor OS memory control that allows programs to read uninitialized memory. A recent example for me was

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-49340831980913218832011-06-22T10:10:00.000-07:002011-06-22T11:04:20.411-07:00

Google is hiring and there are lots of opportunities to do Machine Learning-related work here. Kevin Murphy is applying Bayesian methods to video recommendation, Andrew Ng is working on a neural network that can run on millions of cores, and that's just the tip of the iceberg that I've discovered working here for last 3 months.There is machine learning work in both "researcher" and "engineer"

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com10tag:blogger.com,1999:blog-10560800.post-33906423479759773362011-04-30T13:47:00.000-07:002011-05-01T10:30:22.434-07:00

Five years ago I ran some queries on Google Scholar to see trends on the number of papers that mention particular phrase. The number of hits for each year was divided by the number of hits for "machine learning". Back then it looked like NN's started gaining in popularity with invention of back-propagation in 1980's, peaked in 1993 and went downhill from there.Since then, there's been several

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com7tag:blogger.com,1999:blog-10560800.post-60601209416587544842011-04-29T22:28:00.001-07:002011-04-29T22:56:53.789-07:00

I just noticed that Justin Domke has a blog -- He's one of the strongest researchers in the field of graphical models. I first came across his dissertation when looking for a way to improve loopy-Belief Propagation based training. His thesis gives one such idea -- instead of maximizing the fit of an intractable model, and using BP as intermediate step, maximize the fit of BP marginals directly.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-10419513453998671602011-03-13T21:13:00.000-07:002011-03-13T22:00:04.334-07:00

I've accepted an offer from Google and will be joining their Tesseract team next week.I first got interested in OCR when I faced a project at my previous job involving OCR of outdoor scenes and found it to be a very complex task, yet highly rewarding because it's easy to make incremental progress and see your learners working.Current state-of-the-art OCR tools are not at human level of reading,

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com10tag:blogger.com,1999:blog-10560800.post-5746496796180683102011-03-05T03:52:00.000-08:002011-03-05T04:35:22.222-08:00

Maximum independent set, or "maximum stable" set is one of classical NP-complete problems described in Richard Karp's 1972 paper "Reducibility Among Combinatorial Problems". Other NP-complete problems often have a simple reduction to it, for instance, p.3 of Tony Jebara's "MAP Estimation, Message Passing, and Perfect Graphs" shows how MAP inference in an arbitrary MRF reduces to Maximum Weight

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-14774976175351781742011-03-03T16:38:00.000-08:002011-03-13T22:13:33.862-07:00

A recent discussion on stackoverflow brought up the issue of results of floating point arithmetic being non-reproducibleA reader asked what one could do to guarantee that result of floating point computation is always the same, and Daniel Lichtblau, a veteran developer at the kernel group of WRI replied that "it is impossible with current hardware and software"One problem is that IEEE 754

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-44570509532100760462011-02-21T15:52:00.000-08:002011-02-24T17:54:44.770-08:00

Today I got Google Alert today on the following pending patent -- Belief Propagation for Generalized Matching. I like to stay up on Belief Propagation literature, so I took a closer look. The PDF linked gives a fairly detailed explanation of belief propagation for solving matching problems, including pseudocode which is very detailed, looking like an excerpt of a C program. Appendix A seems to

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-24719230411237745512011-02-20T17:20:00.000-08:002011-02-21T01:05:43.716-08:00

With regular distributive law you can do things like$$\sum_{x_1,x_2,x_3} \exp(x_1 + x_2 + x_3)=\sum_{x_1} \exp x_1 \sum_{x_2} \exp x_2 \sum_{x_3} \exp x_3$$This breaks the original large sum into 3 small sums which can be computed independently.A more realistic scenario requires factorization into overlapping parts. For instance take the following$$\sum_{x1,x2,x3,x_4,x_5} \exp(x_1 x_2 + x_2 x_3 +

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-44014268610350570272011-02-15T14:06:00.000-08:002011-02-15T23:31:34.872-08:00

Suppose we want to count the number of independent sets in a graph below.There are 9 independent sets.Because the graphs are disjoint we could simplify the task by counting graphs in each connected component separately and multiplying the resultVariational approach is one way of extending this decomposition idea to connected graphs.Consider the problem of counting independent sets in the

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com11tag:blogger.com,1999:blog-10560800.post-17941838378465142022011-02-09T20:51:00.000-08:002011-02-10T10:37:03.085-08:00

There's a neat connection between Cholesky factorization and graph triangulations -- graph corresponding to Cholesky factorization of a sparse matrix is precisely the triangulation of the sparse matrix (when viewed as a graph) using canonical elimination ordering.Here's an example of a matrix (corresponding to the graph with black edges only), and its Cholesky factorization. We triangulate the

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-11043719896585946052011-01-21T22:13:00.000-08:002011-01-23T18:57:17.949-08:00

Here's a 367 vertex Apollonian Network and its Junction Tree (aka Tree Decomposition)A Junction Tree provides an efficient data structure to do exact probabilistic inference. In the context of traditional graph algorithms, it is known as the "Tree Decomposition". The amount of structure apparent from the junction tree shows that problems structured as Apollonian Networks have very low

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-29431570588438110362011-01-14T19:56:00.000-08:002011-01-14T20:18:05.652-08:00

Here's a page linking 65 attempts of resolving P vs NP problem. A couple of papers were published in peer-reviewed journals or conferences, while most are "arxiv" published. Some statistics:35 prove P=NP28 prove P!=NP2 prove it can go either way (undecidable)

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com5tag:blogger.com,1999:blog-10560800.post-2904097030897368942011-01-08T14:20:00.000-08:002011-01-09T08:55:53.308-08:00

First programmers wrote in machine code and assemblers simplified this task significantly by letting them give algorithms at a higher level. I still find stacks of punch cards like below in my St.Petersburg homeWouldn't it be great if we could extend this idea further and have the computer compile the problem into machine code?Actually, we already have such tools restricted to various versions of

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-80587839649907948662011-01-02T13:55:00.000-08:002011-01-02T19:48:07.636-08:00

Here's a tool (in Mathematica) to help visualize the process of constructing a Junction Tree. wrong link fixedClicking on vertices corresponds to vertex elimination and each click creates a new bag. After all vertices are eliminated, junction tree is created by removing redundant bags and taking the maximum spanning tree with size of the separators defining the weight of each edge as the junction

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-80118176211232626112011-01-01T22:13:00.000-08:002011-01-01T22:39:50.428-08:00

Here are some links to start the new year on a light noteHinged tesselation Statistics-related Cartoons on stats.SEMemorable math paper titles, like Coxeter's "My Graph" (about Coxeter graph). Memorable Computer Science paper titles. It includes a series of papers "Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire", and "Seee More through Lenses than Bananas.". Apparently

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-75941130479424260662010-12-21T21:43:00.001-08:002010-12-22T11:37:25.413-08:00

Suppose we'd like to visualize a set of joint probabilities realizable by distributions of 3 binary variables.It is a 7 dimensional regular simplex, and we could draw a lower dimensional section of it. There's an infinite number of sections, but there aren't that many *interesting* ones.As an example, suppose we want to find a good 2 dimensional section of a 3 dimensional regular simplex. It

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-46834102149639711902010-12-15T16:05:00.000-08:002010-12-16T23:17:29.578-08:00

One approach to solving hard problems is to break them into computationally efficient parts.For instance, Globerson/Jaakkola do approximate counting by decomposing graph into planar graphs. In each part, the problem reduces to perfect matchings which can be solved efficiently on planar graph. Bouchet/Jordan decompose counting bipartite perfect matchings into much easier problems that count

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com6tag:blogger.com,1999:blog-10560800.post-68520521764365813872010-12-12T13:10:00.000-08:002010-12-14T11:45:22.469-08:00

A few "connection to other fields" papers I found interestingVariational Inference over Combinatorial Spaces. Alexandre Bouchard-Cote, Michael Jordan:Extend variational formulation to upper bound the number of combinatorial structures with global constraints, like perfect matchings and TSP tours. The trick is to represent the space of structures as an intersection of spaces that are tractable to

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-73517213752074814232010-12-05T10:02:00.001-08:002010-12-05T10:04:03.503-08:00

Most graphics and formulas that appear in this blog were created with help of Mathematica. I'll keep a separate blog with Mathematica tips.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-57957491694873774352010-12-03T12:21:00.000-08:002010-12-03T13:00:58.951-08:00

Common complaint about current publishing model is that sometimes good papers get rejected. A striking example is that David Lowe's SIFT algorithm was rejected multiple times from vision venues. The author then assumed that vision community is not interested, and applied for patent intended promote it just for industrial applications. As a result, what's arguably the most popular key-point

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com3tag:blogger.com,1999:blog-10560800.post-87459196321937897802010-11-30T12:41:00.000-08:002010-11-30T13:02:42.974-08:00

In order to do exact probabilistic inference on real life network efficiently, one must find a good Tree Decomposition of the network. This process is known as the Junction Tree algorithm. It's a bit hard to visualize the result, but while browsing tree decomposition graph pages on Wikipedia, I got an idea. Instead of bags with variables, we plot it as a collection of colored strips where each

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-45475115963212575662010-11-28T09:20:00.000-08:002010-11-28T09:55:33.933-08:00

Link here.If you never heard of the journal "Machine Learning", it used to be number 1 ranked journal in ML, until board of editors resigned and founded the Journal of Machine Learning Research, possibly the greatest success story in open-access publishing

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-23831518270306951772010-11-19T13:55:00.000-08:002010-11-19T13:57:50.370-08:00

I just came across kaggle.com which is a platform for "s a platform for data prediction competitions." From brief glance, it seems the goal is to automate Netflix-prize-like competitions. It seems four competitions are currently active

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-62538038963393566762010-11-17T22:25:00.000-08:002010-11-17T22:41:25.010-08:00

Ulrich Bodenhofer has made some nifty SVM visualization code. One is Mathematica notebooks that takes libSVM model files and visualizes them, another one generates plots of Bayes optimal classifier on some synthetic 2d problems

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-21179915232510967552010-11-02T23:24:00.000-07:002011-01-01T22:42:18.759-08:00

Searching for "NIPS" in google blog-search mainly produces posts about nipples, my "CRF" subscription on delicious recently flooded me with links about "Chronic Renal Failure" (apparently a common feline ailment), and you can guess what I got when trying to find Bill Sutherlands "Beautiful Models" book (it's about statistical physics models).Other unfortunate name choices -- hard-ass (hard as

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-66638434009796960732010-10-28T16:59:00.000-07:002011-04-26T22:23:37.524-07:00

In 1993, R.Holte noted that "Very simple classification rules perform well on most commonly used datasets". His very simple rules are basically binary classifiers that make decision based on value of a single attribute.In the race for latest and greatest classifiers it's easy to forget that many kinds of classification tasks in 1993 still come up today, so conclusions of Holte's paper still

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-60253922889733447762010-10-26T11:03:00.000-07:002010-10-26T20:05:18.989-07:00

Suresh points out that at this year's FOCS, not a single person wanted printed proceedings, whereas few years ago, a third of the audience would ask for printed version.I see a similar shift happening with technical books. Purists say that you can't beat the convenience of browsing a real book, but I say that you can't beat the convenience of having access to all your books wherever you go. In

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-24811848175539762002010-10-22T10:11:00.000-07:002010-10-22T10:14:41.027-07:00

David Mimno fit a Dirichlet-multinomial to ICML papers 2004-2008. Seems like "real world" problems are going strong, while boosting took a serious dive

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-68148694745234985762010-10-13T18:26:00.001-07:002010-10-13T18:26:44.008-07:00

A comment on previous post asked why we need integrals for computer science. One reason is that combinatorial expressions often have representation in terms of integrals. Consider binomial coefficients. We have the following$${n \choose \frac{n+d}{2}}=\frac{1}{2 \pi} \int_{-\pi}^\pi e^{-\mathbb{i} q d} (2\cos q)^n dq$$To see where this comes from, consider that $2 \cos x=e^{-\mathbb{i}\pi}+e^{\

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-4648143417203882222010-10-03T22:11:00.000-07:002010-10-03T22:12:55.190-07:00

Thanks to John Cook for pointing it outTheoretical CS cheat sheet

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com5tag:blogger.com,1999:blog-10560800.post-75619549951711454982010-10-02T10:50:00.000-07:002010-10-02T11:02:06.854-07:00

Terry Tao's article on universality.Stephen Wolfram's speech on future special functions.They give opposing perspective -- in the end of speech Stephen Wolfram argues that most processes in universe are "computationally irreducible" and we have no hope to simulating them accurately, whereas Terry Tao's article gives examples of many things in real world obeying a small set of simple laws

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-41249775648208081622010-09-25T14:34:00.001-07:002010-09-25T15:23:28.783-07:00

Suppose I have an invertible function $f(x)$. In a perfect world, the following holds$$x=f(f^{-1}(x))=f^{-1}(f(x))$$To see what happens in a real world, consider the following$$D=\left(\begin{matrix}1&1 \\\\ 1&0\end{matrix}\right)^{\otimes\ d}$$$$f(\mathbf{x})=D\exp (\mathbf{x} D)$$$f(x)$ maps natural parameters $x$ to mean value parameters $\mu$ in an exponential family over $\{0,1\}^d$Let $d=6,

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-10566416079760531462010-09-24T12:13:00.000-07:002010-09-24T12:26:18.262-07:00

I recently raked the blogosphere for interesting new Machine Learning/math blogs and got a high recall, low precision list of 87, here.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-11320183877753106632010-09-16T16:19:00.000-07:002010-09-16T20:39:15.007-07:00

Suppose X is distributed as n-dimensional Gaussian with 0 mean and concentration matrix $A$ and you need conditional distribution of $P(\mathbf{x}|\mathbf{vx}=\mathbf{0})$ where $\mathbf{v}$ is some unit norm vector. To normalize this density you need to integrate $\exp(-\mathbf{x}'A\mathbf{x})$ over subspace of $\mathbb{R}^n$ orthogonal to $\mathbf{v}$, how do you do it?Take the Dirac delta

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-81383592432581598432010-09-05T11:16:00.000-07:002010-09-08T12:14:41.446-07:00

Foundations of probabilistic inference is often a subject of much disagreement, with some leading Bayesian sometimes going as far as to say that MaxEnt method doesn't make any sense, and the MaxEnt camp picking on the issue of subjectivity.The way I see it, MaxEnt and Bayesian approaches are just different ways of using external information to pick a probability distribution.With Bayesian

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-74731351131980576132010-09-04T13:59:00.000-07:002010-09-04T22:14:38.646-07:00

Suppose we throw a fair coin n times and estimate it's bias by averaging the number of heads observed. What is the squared error of this estimator?Using standard binomial identities we can calculate this quantity exactly$$\sum_{k=0}^n {n\choose k} 2^{-n} (\frac{k}{n}-\frac{1}{2})^2=\frac{1}{4n}$$Another approach is to use the Central Limit Theorem to approximate exact density with a Gaussian.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-37504834999273467672010-08-30T13:43:00.000-07:002010-08-30T13:45:43.402-07:00

By Frank Nielsen, with focus on information geometry, http://blog.informationgeometry.org/

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-81441859297346369682010-08-15T15:35:00.001-07:002010-08-24T20:42:15.966-07:00

After following some discussions on overflow sites, I re-read Shannon/Cover's coverage of method of types, and want to summarize it here because of how useful and underappreciated it is.A type or type class is basically an empirical distribution for a sequence of n independent events. For instance, suppose we flip coin n=2 times. We have 3 type classes, all heads, all tails, one head/one tail.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-10772547239228265752010-08-10T17:59:00.000-07:002010-08-11T20:55:38.778-07:00

As I discovered recently, stack exchanges can be pretty fun, informative (and time consuming!) way to discuss issues related to machine learning. Here are some relevant ones I foundStatistical AnalysisMathNatural Language ProcessingHigh-level math (non-research level questions will be closed)There's also stack exchange for Machine Learning and Computer Vision that are in proposal phase. Any

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-21686464305960485012009-08-21T18:04:00.001-07:002011-04-26T22:24:50.844-07:00

I used the "Robust OCR dataset" below to make a system for reading runner bibs in video. Standard ML techniques give fairly good results without much tweaking -- AdaBoost with stumps to go through all connected components (in thresholded image) and generate potential candidates, SVM/Gaussian kernel to classify those candidates into digits. Here's a screenshot and a video of this system in

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com6tag:blogger.com,1999:blog-10560800.post-20961762255399538402009-08-05T15:02:00.001-07:002009-08-05T15:34:34.559-07:00

I've collected this dataset for a project that involves automatically reading bibs in pictures of marathons and other races. This dataset is larger than robust-reading dataset of ICDAR 2003 competition with about 20k digits and more uniform because it's digits-only. I believe it is more challenging than the MNIST digit recognition dataset.I'm now making it publicly available in hopes of

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com11tag:blogger.com,1999:blog-10560800.post-30693152590599954312009-07-12T14:57:00.000-07:002009-07-12T14:59:44.373-07:00

This seems to be a fairly comprehensive vision bibliography. I found a few articles on text localization there that I didn't find through scholar/citation followinghttp://www.visionbib.com/bibliography/contents.html

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-63900057941169509032008-04-01T14:05:00.000-07:002009-04-17T15:22:22.372-07:00

I've started a new job at MyStrands.com, looking at mining financial data. I also applied to Wolfram Research at the same time, but they were too slow to reply. As part of WRI application, I've put together some prior work, I think these are good examples of the things you can use Mathematica forNotebookWeb version

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-47978903060630493652008-02-11T00:10:00.000-08:002008-02-11T00:14:54.500-08:00

Here's a side to side comparison of 150 vs. 300 vs. 600 DPI scan viewed at 400% magnification. On a local library scanner, 600 DPI black-and-white takes twice as slow to scan as 300 DPI, without significant improvement in quality

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-57053348076726964652008-02-06T14:25:00.000-08:002008-02-06T14:50:31.189-08:00

Newton once wrote to Hooke: "If I have seen further it is by standing on the shoulders of giants". It's true nowdays more than ever, and since there's such a huge volume of literature that is electronically searchable, the hard part isn't finding previous work, but remembering where you have found it.Here's the strategy I use, which relies mainly on CiteULike and Google Desktop, what are some

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com15tag:blogger.com,1999:blog-10560800.post-34408764115705556452008-02-01T15:42:00.000-08:002008-02-01T15:51:08.739-08:00

Pi comes up in the most unexpected places. Here's an application to walk counting that involves it.Suppose you have a chain of length n. How many walks of length k are there on the chain? For instance, for a chain of length 5, there are 5 paths of length 0 (start at each vertex and don't go anywhere), 8 of length 1 (traverse each edge either left-right or right-left), 14 of length 2 (8 walks that

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-28912868551195883832008-01-28T16:34:00.000-08:002008-02-13T01:30:31.785-08:00

A few weeks ago , I brought up a Russian CAPTCHA that asks to find resistance between end-points in a version of Wheatstone bridge in order to access a forumThere's been some discussion of this problem on Reasonable Deviations blog, with Vladimir Zacharov writing up a general solution for this circuit using two Kirchhoff's (Kirkhoff) laws and one Ohm's law directly . For larger circuits, this

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-70626726226261895132008-01-25T16:33:00.000-08:002008-01-27T13:28:28.982-08:00

I was going to post this to Wolfram's Demonstrations website, but then I realized it doesn't fit some technical format limitations, so I'm posting it here.notebookIt's a demonstration of Expectation-Maximization algorithm, you need Mathematica or free Mathematica Player to run it.Expectation-Maximization algorithm tries to find centers of clusters in the data. It first assigns each point to some

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-33921067166574419582008-01-17T14:39:00.000-08:002008-01-17T14:45:59.231-08:00

Maximum entropy principle is the idea that we should should pick a distribution maximizing entropy subject to certain constraints. Many known distributions come out in this way, such as Gamma, Poisson, Normal, Beta, Exponential, Geometric, Cauchy, log-normal, and others. In fact, there's a close connection between maxent distributions and exponential families -- Darmois-Koopman-Pitman theorem

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-90794436872488298412007-12-26T11:39:00.000-08:002007-12-26T11:47:41.201-08:00

Here's an innovative CAPTCHA I came across when trying to register for a forum at http://lib.mipt.ru/?spage=reg_userYou have to enter resistance between A and B in the diagram below. Can you do it?

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com6tag:blogger.com,1999:blog-10560800.post-17808783007948403712007-12-17T20:08:00.000-08:002007-12-18T14:42:37.617-08:00

Stochastic matrices are easy to get -- just normalize the rows. Doubly stochastic matrices require more work -- simply normalizing columns/rows will not converge may take few dozen iterations to converge. One approach that works is to do constrained optimization, finding closest (in least squared sense) doubly stochastic matrix to given matrix. Another approach is to start with a permutation

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com6tag:blogger.com,1999:blog-10560800.post-80855765400903630322007-11-25T18:01:00.000-08:002010-07-29T16:17:57.611-07:00

Consider a distribution over 6 binary random variables captured by the structure below.Red represents -1 potential, green represents +1 potential. More precisely, it's the diagram of the following model, where x's are {-1,1}-valuedWe'd like to find the odds of x1 being 1. You could do it by dividing sum of potentials over labellings with x1=1 by corresponding sum over labellings with x1=-1.In a

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com3tag:blogger.com,1999:blog-10560800.post-49852269088600827352007-11-09T23:47:00.000-08:002008-01-24T13:24:11.946-08:00

Ising Models are important for Machine Learning because they are well-studied physical counter-parts of binary valued undirected graphical models. Belief Propagation in such models is equivalent to iteration of the Bethe-Pieirls fixed point equations. Recently Michael Chertkov and Vladimir Chernyak formulated an expression that gave exact expression for the partition function in terms of a local

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-38737957881469743682007-11-08T17:40:00.000-08:002007-11-09T18:36:47.648-08:00

As mentioned in another post belief propagation is a an important algorithm both in probabilistic inference, and statistical thermodynamics. An interesting, and open question both in physics and graphical inference communities is finding the rate at which Belief-Propagation converges for various graphs. Knowing that belief propagation approximately converges after k iterations, would mean that

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-86419145984391662692007-11-04T14:50:00.000-08:002007-11-04T14:59:53.003-08:00

Congress recently passed a bill that requires all NIH funded researchers make their papers publically available. If there's no veto, this would require NIH researchers to upload final versions of their papers to PubMed Centrallink 1link 2

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-20403785241507231182007-11-02T03:42:00.000-07:002007-11-02T03:44:35.269-07:00

Software Libraries and Their Reuse: Entropy, Kolmogorov Complexity, and Zipf’s Law

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-47955712899293724862007-10-30T01:37:00.000-07:002007-10-30T01:45:44.220-07:00

A promising proof of universality of a 2,3 Turing machine (announced here) seems to have stirred some controversy. Vaughan Pratt's post and Todd Rowland's reply

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-79281837867958272922007-10-26T16:56:00.000-07:002007-10-26T19:11:57.387-07:00

I've recently gone to Northwest Probability Seminar and one topic that came up was Thorp shuffling:Take a deck of cards, split it in 2 evenly, then proceed to riffle them by dropping card from stack x, followed by card from the other stack, where x could be stack 1 or stack 2 with equal probability. So if we have 4 cards, numbered 1..4, then 1234 will become one of 1324,1342,3124,2142 after one

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-21848844129736190172007-09-05T12:44:00.000-07:002007-09-05T14:32:03.839-07:00

There seems to be a difference in definitions of "positive-definite" between pure and applied math literature. For instance:Linear algebra textbook (Horn and Johnson, 1990):x^*Ax>0 for all non-zero xx^* refers to conjugate transposeMachine Learning textbook (Bishop, 2005)x'Ax>0 for all non-zero x (real domain is implied)MathworldRe(x^*Ax)>0The main difference in the definitions is that when

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-50502571424716463722007-08-13T19:27:00.001-07:002007-08-13T20:03:48.905-07:00

Here's a problem that could be solved using either analysis or information theory, which approach do you think is easier?Suppose X_1,X_2,... is an infinite sequence of IID random variables. Let E be an event that's shift invariant, ie, if you take the set of all sequences of Xi's that comprise the event, and shift each sequence, you'll have the same set. For instance "Xi's form a periodic

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-67630120695935729872007-08-07T18:06:00.000-07:002007-08-07T23:31:28.024-07:00

Many people have seen a copy of this tongue-in-cheek email on proof techniques that's been circulating since the 80's. It' funny because it's true, and here is a couple examples from machine learning literatureProof by reference to inaccessible literature, The author cites a simple corollary of a theorem to be found in a privately circulated memoir of the Slovenian Philological Society, 1883.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-82641325334694637332007-08-01T12:20:00.000-07:002007-08-01T13:50:01.653-07:00

Looking at the papers from this summer's machine learning conferences(AAAI, UAI, IJCAI, ICML,COLT) it seems like there have been a lot of papers on L1 regularization this year. There are at least 3 papers on L1 regularization for structure learning by Koller, Wainwright, Murphy, several papers on minimizing l1 regularized log likelihood by Keerthi, Boyd, Gallen Andrew. A couple of groups are

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-33556805461825781072007-07-29T12:20:00.000-07:002007-07-29T12:27:07.080-07:00

David Oranshak is using evolutionary algorithms to create aesthetically pleasing artworkAlso, "human vs computer" design awards for 2007

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-24499278373933110862007-07-28T11:46:00.000-07:002007-07-28T11:50:22.037-07:00

The top performing system exhibited better performance than human evaluators when matching faces under varying lighting conditions in NIST large scale face recognition benchmark.See Figure 8 inhttp://face.nist.gov/frvt/frvt2006/FRVT2006andICE2006LargeScaleReport.pdf

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-72078595305523212992007-07-26T13:00:00.000-07:002007-07-26T18:07:47.632-07:00

Suppose A,B are nxn matrices, and x is an nx1 vector. Need to compute M1M2...Mm x where each Mi is either A or B. The straightforward approach is to start multiplying from the right, which takes O(n^2 m) operations. Is it possible to have an O(n^2 m) algorithm that solves the problem above, but has a lower time complexity than the baseline?The lower bound is O(n^2+m) because that's how long it

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-61656494190363609892007-07-19T22:27:00.001-07:002007-07-19T22:31:03.403-07:00

Looks like UCI website has been updated with some new data, including some sequential/relational datasetshttp://archive.ics.uci.edu/beta/datasets.html

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-67100651419911840692007-07-13T13:08:00.001-07:002007-07-19T20:05:51.289-07:00

I've across the following question recently -- suppose you have a fully specified Markov-1 Hidden Markov Model with binary observations {Xi} and binary states {Yi}. You observe X1,...,Xn and have to predict Yn. Is the Bayes optimal classifier linear? Empirically, the answer seems to be "yes", but it's not clear to how show it.---Update actually looks like they are not linear in general, which can

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-90731911456787851062007-07-09T22:09:00.000-07:002007-07-09T22:26:42.543-07:00

ILP 07 had a discussion panel on next 10 years of structured learning. The video is now posted on Video LecturesHere are couple of things that caught my eye1. Bernhard Pfahringer -- someone should work on a centralized repository like http://www.kernel-machines.org/, except for structured learning2. Thomas Dietterich -- need a way to combine benefits of high-performance discriminative learning

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-53269568992623669252007-07-03T17:01:00.000-07:002007-07-03T19:20:51.624-07:00

People often fit the model to data by minimizing J(w)+w'Bw where J is the objective function and B is some matrix. Normally people use diagonal or symmetric positive definite matrices for B, but what happens if you use other types?Here's a Mathematica notebook using Manipulate functionality to let you visualize the shrinkage that happens with different matrices, assuming J(w)'s Hessian is the

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-45348840968123517792007-06-28T14:39:00.000-07:002007-06-28T19:07:08.731-07:00

Here are a few that caught my eye:Scalable Training of L1-regularized Log-linear ModelsThe main idea is to do L-BFGS in an orthant where the gradient of the L1 loss doesn't change. Each time BFGS tries to step out of that orthant, project it's new point on the old orthant, and figure out the new orthant to exploreDiscriminative Learning for Differing Training and Test DistributionsIn addition to

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-25401688831653074442007-06-27T00:26:00.000-07:002007-06-27T16:43:16.013-07:00

I found a large number of machine learning related patent applications by doing a few queries on http://www.freepatentsonline.com/Here are a couple that caught my eye:Logistic regression (A machine implemented system that facilitates maximizing probabilities)Boosting (A computer-implemented process for using feature selection to obtain a strong classifier from a combination of weak classifiers)

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com8tag:blogger.com,1999:blog-10560800.post-17436591711425556262007-06-12T16:52:00.000-07:002007-06-13T09:57:21.200-07:00

Suppose you want to predict binary y given x. You fit a conditional probability model to data and form a classifier by thresholding on 0.5. How should you fit that distribution?Traditionally people do it by minimizing log-loss on data, which is equivalent to maximum likelihood estimation, but that has the property of recovering the conditional distribution exactly with enough data/modelling

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-32148389396737853232007-05-03T16:22:00.000-07:002007-05-03T16:29:24.556-07:00

Mathematica 6.0 is out, touted "The most important advance in the 20-year history of Mathematica"Among the highly anticipated features is the support for joysticks/gamepads (an XBox version is surely to follow shortly), and the ability to use freehand doodles instead of mathematical symbols in equations (handy when you run out of latin/greek alphabets in your equations)

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-1148020665166793182006-05-18T23:25:00.000-07:002011-09-07T11:59:48.592-07:00

There are some counter-intuitive things happening as dimension increases.For instance consider unit sphere (radius 1)If you go from 2 dimensions to 3 dimensions, the volume increases (Pi to 4/3 pi). Similar increase happens when you go to 3,4,5 dimensions. But then the volume starts to decrease. Eventually it decreases all the way to 0.Now consider a cube of width 1. As dimension increases, the

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com7tag:blogger.com,1999:blog-10560800.post-1147208485794625442006-05-09T13:50:00.000-07:002007-06-12T11:30:57.158-07:00

Concepts of positive definite matrices and positive definite operators often come up in Machine Learning, here are some movies I made to visualize linear transformations corresponding to some common types of matrices:Symmetric and positive definiteSymmetric positive definite matrices can be thought of matrices that stretch or shrink the circle along the eigenvectors (marked in blue).The angle

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-1146364961893729792006-04-29T16:39:00.000-07:002007-06-13T11:26:12.856-07:00

There's often confusion as to the nature of the differences between Logistic Regression and Naive Bayes Classifier. One way to look at it is that Logistic Regression and NBC consider the same hypothesis space, but use different loss functions, which leads to different models for some datasets. To see that both logistic regression and naive bayes classifier consider the same hypothesis space we

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-1145334628428202262006-04-17T21:29:00.000-07:002007-08-24T12:09:58.685-07:00

Here is an succinct derivation of probability calculus by Skillings (appeared in Appendix of his MaxEnt 2005 "Bayesics" article)

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-1143847017814312072006-03-31T14:25:00.000-08:002006-03-31T15:30:16.060-08:00

I needed to refresh some Graphical Models basics, and here are lecture notes I found usefulJeff Bilmes' Graphical Models course , gives detailed introduction on Lauritzen's results, proof of Hammersley-Clifford results (look in "old scribes" section for notes)Kevin Murphy's Graphical Models course , good description of I-mapsSam Roweis' Graphical Models course , good introduction on exponential

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-1143058282479080712006-03-22T10:22:00.000-08:002007-06-12T11:42:03.353-07:00

If you've used cell phones to send text messages you probably know about their auto-complete feature. For those that haven't -- each digit corresponds to 3 or 4 letters, you enter the digits consistent with your word, and it tries to guess which word you meant. For instance you enter "43556", and it will automatically guess "hello". But if you enter "785" to mean "SVM", it'll probably guess "run"

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-1141457710901106062006-03-03T23:23:00.000-08:002006-03-03T23:35:10.913-08:00

Learning from presentation or slides can be a much easier than from papers. I was reminded of that when I ran across Martin Wainwright's excellent class on Graphical Models. It has lecture notes *and* videos online. Another large repository of machine learning related videos is the repository organized by the Pascal Network. You may run into a couple of bugs, but Sebastjan Mislej (sebastjan dot

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com6tag:blogger.com,1999:blog-10560800.post-1134791971101653192005-12-16T17:07:00.000-08:002007-06-12T11:50:40.983-07:00

In a previous post I brought up the 1983 Machine Learning workshop which featured "33 papers", and it was the follow up to the 1980 Machine Learning workshop. By contrast, NIPS 2005 had 28 workshops and is just one of several international annual Machine Learning Conferences. You can see how the field grew by looking at the distribution of publication dates for articles containing phrase "machine

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com8tag:blogger.com,1999:blog-10560800.post-1134465378909195432005-12-13T01:07:00.000-08:002005-12-13T10:41:33.963-08:00

At the NIPS "Towards Human-Level AI" workshop one of the messages was that perhaps we should first try to achieve rat-level AI, and go from there.But maybe instead we should start by achieving pigeon-level AI. Someone has already measured pigeon performance at discriminating between painting styles

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-1134427146371411312005-12-12T14:11:00.000-08:002005-12-12T14:39:06.386-08:00

Here's what Aleks Jakulin managed to dig up in Google newsgroup archives"1983 INTERNATIONAL MACHINE LEARNING WORKSHOP: AN INFORMAL REPORT"linkSome snippetsHere's Tom Dietterich again: ``I was surprised that you summarized the workshopin terms of an "incremental" theme. I don't think incremental-nessis particularly important--especially for expert system work."So the language analysis problem has

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com0tag:blogger.com,1999:blog-10560800.post-1131854472048878592005-11-12T12:48:00.000-08:002007-06-13T08:49:42.226-07:00

The Key Theorem of the Learning Theory is what Vapnik calls his theorem (for instance here) that gives the necessary conditions for the Empirical Risk Minimization principle to be consistent.The important equation is the following:Here I'll explain the parts of the formula using the following motivating example:Suppose our true distribution is a normal distribution with unit variance, centered at

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-1131080660823841282005-11-03T20:55:00.000-08:002005-11-03T21:07:16.406-08:00

A google search reveals the following preprints associated with NIPS 2005:Lafferty, Blei, Correlated Topic ModelsLafferty, Wasserman, Rodeo - Sparse Nonparametric Regression in High DimensionsTong Zhang and Rie K. Ando. Analysis of Spectral Kernel Design based Semi-supervised Learning.Philipp Häfliger, et al, AER Building Blocks for Multi-Layer Multi-Chip Neuromorphic Vision SystemsPaninski, L. -

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com4tag:blogger.com,1999:blog-10560800.post-1129497061926174222005-10-16T13:04:00.000-07:002005-10-16T14:11:01.946-07:00

Full texts of recent machine learning papers are usually freely available on the web (ie, google). That's often not the case for related fields like statistics/math/econometrics, so one has to rely on some indexing service that links into commercial article databases. There are hundreds of such indexing services. I tend to use the following indexes, which I think have the highest recall for ML/

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com3tag:blogger.com,1999:blog-10560800.post-1129423810262134392005-10-15T17:28:00.000-07:002005-10-15T17:50:10.276-07:00

Here are some useful Mathematica training notebooks I came across.Mathematica Training -- Notebooks from a 2 day Mathematica courseNKS summer school -- some notebooks from the "New Kind of Science" summer school intro to MathematicaProgramming Paradigms via Mathematica - notebooks from a course developed by Neidinger and Swallow

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com2tag:blogger.com,1999:blog-10560800.post-1110516615897538082005-03-10T14:12:00.000-08:002007-10-21T22:00:13.490-07:00

I found a book scanner in our library's basement and decided to put it to good use by scanning some hard-to-find-online references on foundations of Bayesianism. "Algebra of Probable Inference" by Cox, 1961 (aka, Why everyone should be a Bayesian). Demonstrates a functional derivation of probability theory as the unique extension of Boolean Algebra. "Why I'm not a Bayesian" by Clark Glymour,

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com7tag:blogger.com,1999:blog-10560800.post-1110054822076170302005-03-05T12:19:00.000-08:002005-03-05T15:07:40.826-08:00

I've noticed some people I know starting to use CiteULike recently. Services like CiteULike are generally a good development because they increase efficiency of research: usually people share bibliography through bibliography section of their published articles, but a publication can take years to become accessible.The immediate advantage of CiteULike is that it can fill Bib details in for you.

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com1tag:blogger.com,1999:blog-10560800.post-1109305629081710432005-03-04T00:00:00.000-08:002005-03-04T00:08:46.433-08:00

Here are some journals to keep an eye on, along with their RSS feeds. I picked out the list by seeing where my favourite Machine Learning/stats papers came from: Journal of Machine Learning Research (web, rss) The journal for machine learning publications. A better and freer replacement to "Machine Learning" journal -- here's some history IEEE Transactions on Pattern Analysis and Machine

Yaroslav Bulatovhttp://www.blogger.com/profile/06139256691290554110noreply@blogger.com5