Implementing a MultiClass Bayes Classifier (a Generative Model) with Gaussian Class-conditional Densities in Python

The following problems appeared as a project in the edX course ColumbiaX: CSMM.102x Machine Learning. The following problem description is taken from the course project itself.

im1.png

The following figure shows a first few rows of a sample training dataset with d=2 dimensions and K=5 class labels.

x1 x2 y
0 5.731453 4.517417 4
1 -0.743516 2.157767 3
2 5.745734 1.205449 1
3 -0.322106 -0.183691 0
4 2.200057 -1.658819 3

The following figure shows how the Bayesian Classifier can be first trained to estimate the Prior and the Class Conditional Gaussian Densities (as a Generative Model) and then using Bayes Rule the Posterior Probabilities for the class labels can be predicted on the new test dataset.

f1.png

The following figures show the Gaussian class conditional densities learnt from the training dataset with MLE.

 ccd.png

ccd3d.png
The following figure shows the predicted posterior probabilities on the test dataset.

probs.png
The following figure shows the decision boundaries with the trained classifier on the same 2-D test dataset. As can be seen, since the the covariance matrices of the class conditional Gaussians were not shared (unlike as in the case of Linear Discriminant Analysis), the decision boundaries are non-linear and quite complex.
b0.png

The following figures show the decision boundaries learnt with the Bayesian Classsfier trained and tested on different sample datasets respectively.

 b2b3b5b6b7b8b9b10
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s