My First Kaggle Competition (Part Two)

This is the second part in a two-part series. For part one, click here.

Before I continue I’d like to point out that this is not a tutorial on machine learning or Scikit-Learn of any kind. If you would like to read more about those topics, I’ll suggest selecting a link or two in the Resources dropdown in the top right.

Now, we get to the good stuff- the code. Or, at least a summary of the code. There is too much to go line by line, so I’ll just point out the significant parts. Open up the code here and follow along.

Now, I had never done one of these Kaggle competitions so I wasn’t sure at first what to do. I spent some time browsing over some of the early submissions and tutorials to determine a good approach. I looked at everything from simple predictions that were based on point differential to basic stats such as shooting and turnover percentage to some serious moneyball type stuff. I read about people such as Jeff Sagarin and Dean Oliver and terms such as the Four Factors, Elo Rating, and Offensive Efficiency popped up in several articles.

After playing around and experimental with a few methods I eventually decided to use the same metrics that are used in my previous project, the NCAA Stats Webscaper. I decided on the following metrics (which became the feature set for the machine learning model):

• The Four Factors
• Player Impact Estimate (PIE)
• Offensive Efficiency
• Defensive Efficiency
• Defensive Rebounding Percentage
• Assist Ratio
• Free Throw Percentage
• Score Differential
• Rating Percentage Index (RPI)
• Tournament Seed
• Win Percentage

Using this feature set, my model did a horrible job at predicting upsets (only a couple) and consequently performed terribly. I should have known that a March Madness tournament with very few upsets is just not that likely (you can always count on several), and this past tournament there were a lot. Actually, my overall rank in the competition wasn’t that bad. I placed 498 out of 934 teams, which is not terrible. However, I used the same submission to fill out my annual office bracket pool, and that failed with flying colors.

After the submission deadline and while the tournament was playing out, I did a little reading and continued to tinker with my model. After some research I added

• Adjusted Offensive Efficiency
• Adjusted Defensive Efficiency
• Adjusted Efficiency Margin
• Offensive Rebound to Turnover Margin

I found that adding these four features made for a much better performing model, predicting many of the actual upsets.

The first part leads to creating a vector containing all the features for each team in every season from 2003-2018:

Next, the process is to isolate the winning and losing teams and create two new datasets with an added result column: one is the difference in feature vectors of the winners minus losers with a result of “1”; the other is losers minus winners with a result of “0”.

Finally, concatenate the winners and losers on top of each other and sort by season:

What we end up with is a 21 by 1962 dataframe. The columns are the 17 features, the Result column, and three extra columns for Season, Winning Team ID, and Losing Team ID. Each row is a simulated “matchup” between tournament teams.

The second part covers the training, evaluation, and testing of the machine learning models, and the creation of submission file and more readible file that could be used to create an unbeatable bracket for your office pool.

The first step is separating the dataset into training and test data:

y = prediction_dataset['result']
X = prediction_dataset.loc[:, :'WinPCT']
train_IDs = prediction_dataset.loc[:, 'Season':]

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3, random_state=1, stratify=y)

I chose to go with with a 70/30 split of training to test data, which seemed to work considering the amount of data I had.

The next step is to initialize the six different classifiers:

• Logistic Regression
• K Nearest Neighbors
• XGBoost
• Decision Tree
• Random Forest
• Gradient Boosting

and evaluate them using a grid search cross validation. Without going into too much detail, the Logistic Regression classifier peformed the best, returning a test accuracy of 79.80%. I didn’t realize this at the time, but it turns out I made a significant error in evaluating the classifiers. I used the training set in the grid search cross validation. What I should have done was have a validation set in addition to the training and test sets. (Not so much of an independent evaluation, is it?)

Anyway, after fitting the model, I create a prediction dataset to simulate every possible match up of teams in the 2018 March Madness tournament:

The three numbers in the ID column are the year of the tournament, the team #1, and team #2. The number in the Pred column is the probability that team #1 beats team #2. The format is required by Kaggle and doesn’t really help if you want to create a bracket.

The final piece of the code matches up the team IDs with team names and generates readable predictions, such as “Alabama beats Arizona: 0.563389”. This doesn’t necessarily guarantee that Alabama plays Arizona, but if they did there is 56% probability that Alabama wins.

So that’s it, my first Kaggle competition. I have to admit I jumped into this knowing very little about supervised learning, Scikit-Learn, and classifier models, learning most of it on the fly. It was a good experience and I will definitely do it again next year. Maybe I will try different features and difference classifiers. Who knows, maybe I try some deep learning models. One thing I do know- I won’t use training data on my cross validation!

Don’t mind me, I’m just rambling.