First, let's import some stuff we'll need: Pandas, the classifier modules from scikit-learn, a few utility functions for sample selection and performance evaluation
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_fscore_support, confusion_matrix
Next, we'll load the dataset that we're working with. Today, we'll be using the white wine component of the wine quality dataset from the UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/Wine+Quality
df = pd.read_csv('winequality-white.csv',sep=';',quotechar='"')
What can we learn about this dataset? It looks like a number of different features, and then a target variable called "quality", which ranges from 3 to 9. The median and 75th percentile values are both 6 out of 9, so we can determine that this class is unbalanced. Indeed, we see that 5 and 6 make up more than 2/3 of the dataset.
df.columns
df['quality'].unique()
df['quality'].describe()
df['quality'].value_counts()
Let's turn that unbalanced ordinal class into an unbalanced binary class, and apply that to the dataset based on the value of the "quality" feature. We want to build a classifier for the best of the best wines, so it's fine that the class is unbalanced. After all, this isn't Lake Woebegone, where every wine is above average...
def isTasty(quality):
if quality >= 7:
return 1
else:
return 0
df['tasty'] = df['quality'].apply(isTasty)
df.columns
df['tasty'].value_counts()
Next, we'll create training and testing subsets that we'll use to train our classifiers. It is best practice to always try train and test your classifiers on different datasets. Here we'll take one-third of the original population and use it for testing, and the other two-thirds will be used for training the classifiers. Note that we can specify a random_state seed in order to get the same results for the same input data if we want to replicate this experiement later.
data = df[['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar','chlorides', 'free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol']]
target = df['tasty']
data_train, data_test, target_train, target_test = train_test_split(data,target,test_size = 0.33,random_state=123)
Did the splitting process produce populations having the right shapes for what we're trying to do? Yes it did!
[subset.shape for subset in [data_train,data_test,target_train,target_test]]
Now we'll use our split population of train and test data to train some classifiers. Today we'll be using a simple decision tree, a Gradient-Boosting classifier, and a Random Forest Classifier, all from scikit-learn
For all classifiers that we'll be using today, we'll hold the max tree depth at 5. This prevents over-fitting of individual trees, and also allows us to remove that as a factor in the performance of the classifier.
simpleTree = DecisionTreeClassifier(max_depth=5)
simpleTree.fit(data_train,target_train)
gbmTree = GradientBoostingClassifier(max_depth=5)
gbmTree.fit(data_train,target_train)
rfTree = RandomForestClassifier(max_depth=5)
rfTree.fit(data_train,target_train)
Now that the classifiers are trained, let's evaluate their performance. We can call the precision_recall_fscore_support function from sklearn.metrics to return, well... the Precision, Recall, F-Score, and Support measures for each classifier, comparing the original target data with the predicted test data
simpleTreePerformance = precision_recall_fscore_support(target_test,simpleTree.predict(data_test))
gbmTreePerformance = precision_recall_fscore_support(target_test,gbmTree.predict(data_test))
rfTreePerformance = precision_recall_fscore_support(target_test,rfTree.predict(data_test))
The object returned by the precision_recall_fscore_support() function is not the prettiest... but we'll solve that later. As you can see, the function returns the precision, recall, f-score, and support metric for each class. Looking at the support metric, we can see that the class-wise composition of the test population is slightly different than the population as a whole. However, it's not crazy unbalanced, so it will not unduly affect the performance metrics.
simpleTreePerformance
gbmTreePerformance
rfTreePerformance
There we go, that's a bit easier to read. We're interested in predicting a positive (1) value for "tasty", so we're mostly concerned with the performance of each classifier in predicting the value for that class, which is the second number in each list for each metric (ie, the Simple Tree Classifier found 0.566 Precision for the positive class, and the Random Forest Classifier found .326 Recall for the positive class)
With each of these metrics, we're looking for a value as close to one (1) as possible. We can see that the Gradient Boosted (GBM) tree generally out-performs the others in correctly classifying tasty wines. The GBM tree also achieved a higher recall for the positive class than the other classifiers. This all being the case, it is clear that the GBM tree classifier was the strongest performer in the cohort.
print('Precision, Recall, Fscore, and Support for each class in simple, gradient boosted, and random forest tree classifiers:'+'\n')
for treeMethod in [simpleTreePerformance,gbmTreePerformance,rfTreePerformance]:
print('Precision: ',treeMethod[0])
print('Recall: ',treeMethod[1])
print('Fscore: ',treeMethod[2])
print('Support: ',treeMethod[3],'\n')
Here's another way to conceptualize classifier performance - Confusion Matrices. Here, we compare the quantity of predicted values against each actual value. For example, for the Simple Tree, we see that out of the testing sample, 1187 un-tasty wines were correctly classified, and 141 tasty wines were correctly classified. However, there were also 108 tasty wines that were erroneously classified as un-tasty, and 181 un-tasty wines that were classified as tasty. Gross!
print('Confusion Matrix for simple, gradient boosted, and random forest tree classifiers:')
print('Simple Tree:\n',confusion_matrix(target_test,simpleTree.predict(data_test)),'\n')
print('Gradient Boosted:\n',confusion_matrix(target_test,gbmTree.predict(data_test)),'\n')
print('Random Forest:\n',confusion_matrix(target_test,rfTree.predict(data_test)))
So, now that we know that the GBM tree is our favored classifier for predicting the tastiness of wines, that begs the question: "what makes a tasty wine?". Luckily, GBM trees produce interpretable results, so we can call the feature_importances method against the GBM tree object and find out which features play the largest role in predicting tastiness.
One important thing to note is that not all machine learning methods produce interpretable results. For example, the random forest classifier is literally a set of multiple decision trees, so it's not easy to say which features played the largest role in determining the outcome of the classification. Neural Networks and Support Vector Machines (SVM) also suffer from this. That's not to say, however, that it cannot be done using local interpretable model explanations (LIME): https://arxiv.org/pdf/1602.04938v1.pdf
gbmTree.feature_importances_
A little hard to read, we can do better!
Indeed, we see that alcohol content and density play the two largest roles in the decision of the GBM classifier, followed by total sulfur diocide and volatile acidity. On the other hand, fixed acidity and residual sugar appear to be less important factors in determining the tastiness of wine. Perhaps if we were looking at red wines, this observation might be different!
print('Feature Importances for GBM tree\n')
for importance,feature in zip(gbmTree.feature_importances_,['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar','chlorides', 'free sulfur dioxide', 'total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol']):
print('{}: {}'.format(feature,importance))