I just completed the Coursera Machine Learning class this week, and enjoyed the experience very much. Let’s get the obvious out of the way: getting a high-quality class, for free, wherever you are, at your own pace, is pretty amazing, and I can put up with a sometimes flaky video player for that. Every quarter in college, I would agonize over what limited number of classes I should take, thinking that I might not be able to take that class ever again once I graduated, and Coursera is awesome for that – now I know I can keep learning forever, with that worry gone. Thank you!
One thing I was hoping to get from this class was a broader perspective on machine learning. What I know from the topic, I learnt from various disparate sources, collecting ideas, algorithms and recipes. As a result, my knowledge was a bit of a hodge-podge. The class really delivered on that front. For the most part, the lectures progressed logically, from linear regression, to logistic regression, neural networks and support vector machines, all in a unified manner: define a cost function, use regularization to compensate for over-fitting, and fit parameters using gradient descent. I really enjoyed the coherence of the progression, which helped seeing the commonalities between all the approaches.
Following that thought, my biggest take-away was the emphasis on over- or under-fitting. I understood the concept before the class, but it wasn’t as prominently on my mind as it is now. This is probably a side-effect of my past experience in optimization and statistics, where the data was easier to visualize, and the goal was mostly to find the optimum fit, potentially leading to fragile solutions which wouldn’t generalize – over-fitting wasn’t a problem I gave much thought. In a space like machine learning, where datasets are too large to get a visual sense of what’s going on, keeping that question in mind is important. Relatedly, I found the discussions on how to diagnose a model to focus efforts extremely valuable: while more data is usually better, there are situations where it won’t help, and again, with large and hard to comprehend datasets, understanding what is potentially going wrong in a model and why is very important to avoid wasting efforts in the wrong direction, or simply figure out a direction to take when stuck.
One aspect I found interesting is that while quite a few of the models discussed have a long history in statistics (linear and logistic regression for instance), there was virtually no mention of statistics in the entire class – no goodness-of-fit statistics, no null hypothesis, no discussion on how parameters are distributed, nothing like that. Instead, it felt much closer to my background in operations research: define the function you are trying to minimize, and minimize it. I am not sure whether this has any implications regarding where statistics are headed, but found it intriguing.
Finally, the other aspect that struck me was the emphasis on linear algebra. In essence, one of the messages of the class was “if you want high performance, express your problem in a vectorized form”. I can understand why, from two perspectives. First, computers are really, really good at dealing with linear algebra operations, and second, expressing a problem with matrices and vectors is typically nicely compact. Coming from operations research, this is something I am fairly comfortable with. At the same time, the explosion of indices, sub- and super-scripts wasn’t the most pleasant part of the class, and I spent an inordinate amount of time in programming homework just trying to figure out if a particular element was a row or column vector, tinkering with transposition until the bloody product would just work. I found myself really missing some hints on what the shape of a particular element was, or early warning that the operation wasn’t going to work. Relatedly, while there is a nice consistency in working in a world where everything is a vector or matrix of floats, I felt slightly disturbed at representing true or false as 1 and 0s (for instance), and missing cleaner functional operations like filter or map.
That’s it – overall, this was time very well spent. I am very glad I did it, and would recommend that class to anyone who is not allergic to math, and wants a good introduction to the topic!
I got interested in the following question lately: given a data set of examples with some continuous-valued features and discrete classes, what’s a good way to reduce the continuous features into a set of discrete values?
What makes this question interesting? One very specific reason is that some machine learning algorithms, like Decision Trees, require discrete features. As a result, potentially informative data has to be discarded. For example, consider the Titanic dataset: we know the age of passengers of the Titanic, or how much they paid for their ticket. To use these features, we would need to reduce them to a set of states, like “Old/Young” or “Cheap/Medium/Expensive” – but how can we determine what states are appropriate, and what values separate them?
More generally, it’s easier to reason about a handful of cases than a continuous variable – and it’s also more convenient computationally to represent information as a finite set states.
So how could we go about identifying a reasonable way to partition a continuous variable into a handful of informative, representative states?
Last week, we had our first Coding Dojo at SFSharp.org, the San Francisco F# group – and it was great! A few people in the group had mentioned that at that point they were already convinced F# was a great language, and that what they wanted was help getting started writing actual code, so I figured this would be a good format to try out.
What I wanted was something fun, something cool people could realistically achieve under 2 hours. I settled for one of the Kaggle introduction problems, a classic of Machine Learning, where the goal is to automatically recognize hand-written digits. I didn’t think it would be fair to just throw people in the shark tank without any guidance, especially for F# beginners, so I prepared a minimal slide deck to explain the problem and data set, and a “guided script”, with hints and language syntax examples.
And… it worked! The attendees were absolutely awesome. We had people from Kaggle, Rdio, and two people who drove all the way from Sacramento; we had beginners and experienced FSharpers – and everybody managed to get a classifier working, from scratch. Having some beers available definitely helped, too.
My favorite part is this one attendee, a F# beginner, who kept going at it after the meeting was over, and posted an algorithm improvement in the comments section of the Meetup a couple days after. Way to go! And given the positive response, we’ll definitely have more of these.
Also wanted to say a huge thanks to Matt Harrington, first for starting this user group back then, and then for still being an incredible supporter of the F# community in SF, in spite of a crazy work schedule. Thanks, Matt!
In our previous post, we began exploring Singular Value Decomposition (SVD) using Math.NET and F#, and showed how this linear algebra technique can be used to “extract” the core information of a dataset and construct a reduced version of the dataset with limited loss of information.
Today, we’ll pursue our excursion in Chapter 14 of Machine Learning in Action, and look at how this can be used to build a collaborative recommendation engine. We’ll follow the approach outlined by the book, starting first with a “naïve” approach, and then using an SVD-based approach. We’ll start from a slightly modified setup from last post, loosely inspired by the Netflix Prize. The full code for the example can be found here on GitHub.
The problem and setup
In the early 2000s, Netflix had an interesting problem. Netflix’s business model was simple: you would subscribe, and for a fixed fee you could watch as many movies from their catalog as you wanted. However, what happened was the following: users would watch all the movies they knew they wanted to watch, and after a while, they would run out of ideas – and rather than search for lesser-known movies, they would leave. As a result, Netflix launched a prize: if you could create a model that could provide users with good recommendations for new movies to watch, you could claim a $1,000,000 prize.
Obviously, we won’t try to replicate the Netflix prize here, if only because the dataset was rather large; 500,000 users and 20,000 movies is a lot of data… We will instead work off a fake, simplified dataset that illustrates some of the key ideas behind collaborative recommendation engines, and how SVD can help in that context. For the sake of clarity, I’ll be erring on the side of extra-verbose.
Our dataset consists of users and movies; a movie can be rated from 1 star (terrible) to 5 stars (awesome). We’ll represent it with a Rating record type, associating a UserId, MovieId, and Rating:
To make our life simpler, and to be able to validate whether “it works”, we’ll imagine a world where only 3 types of movies exist, say, Action, Romance and Documentary – and where people have simple tastes: people either love Action and hate the rest, love Romance or hate the rest, or love Documentaries and hate the rest. We’ll assume that we have only 12 movies in our catalog: 0 to 3 are Action, 4 to 7 Romance, and 8 to 11 Documentary.
Last Thursday, I gave a talk at the Bay.NET user group in Berkeley, introducing F# to C# developers. First off, I have to thank everybody who came – you guys were great, lots of good questions, nice energy, I had a fantastic time! My goal was to highlight why I think F# is awesome, and of course this had to include a Type Provider demo, one of the most amazing features of F# 3.0. So I went ahead, and demoed Tomas Petricek’s [World Bank Type Provider], and Howard Mansell’s R Type Provider – together. The promise of Type Providers is to enable information-rich programming; in this case, we get immediate access to a wealth of data over the internet, in one line of code, entirely discoverable by IntelliSense in Visual Studio - and we can use all the visualization arsenal of R to see what’s going on. Pretty rad.
Rather than just dump the code, I thought it would be fun to turn that demo into a video. The result is a 7 minutes clip, with only minor editing (a few cuts, and I sped up the video x3 because the main point here isn’t how terrible my typing skills are). I think it’s largely self-explanatory, the only points that are worth commenting upon are:
I am using a NuGet package for the R Type Provider that doesn’t officially exist yet. I figured a NuGet package would make that Type Provider more usable, and spent my week-end creating it, but haven’t published it yet. Stay tuned!
The most complex part of the demo is probably R’s syntax from hell. For those of you who don’t know R, it’s a free, open-source statistical package which does amazingly cool things. What you need to know to understand this video is that R is very vector-centric. You can create a vector in R using the syntax myData <- c(1,2,3,4), and combine vectors into what’s called a data frame, essentially a collection of features. The R type provider exposes all R packages and functions through a single static type, aptly named R – so for instance, one can create a R vector from F# by typing let myData = R.c( [|1; 2; 3; 4 |]).
That’s it! Let me know what you think, and if you have comments or questions.
