Implementing a MultiClass Bayes Classifier (a Generative Model) with Gaussian Class-conditional Densities in Python

The following problems appeared as a project in the edX course ColumbiaX: CSMM.102x Machine Learning. The following problem description is taken from the course project itself.


The following figure shows a first few rows of a sample training dataset with d=2 dimensions and K=5 class labels.

x1 x2 y
0 5.731453 4.517417 4
1 -0.743516 2.157767 3
2 5.745734 1.205449 1
3 -0.322106 -0.183691 0
4 2.200057 -1.658819 3

The following figure shows how the Bayesian Classifier can be first trained to estimate the Prior and the Class Conditional Gaussian Densities (as a Generative Model) and then using Bayes Rule the Posterior Probabilities for the class labels can be predicted on the new test dataset.


The following figures show the Gaussian class conditional densities learnt from the training dataset with MLE.


The following figure shows the predicted posterior probabilities on the test dataset.

The following figure shows the decision boundaries with the trained classifier on the same 2-D test dataset. As can be seen, since the the covariance matrices of the class conditional Gaussians were not shared (unlike as in the case of Linear Discriminant Analysis), the decision boundaries are non-linear and quite complex.

The following figures show the decision boundaries learnt with the Bayesian Classsfier trained and tested on different sample datasets respectively.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s