Prerequisites

• DS-GA-1001: Intro to Data Science or its equivalent
• DS-GA-1002: Statistical and Mathematical Methods or its equivalent
• Solid mathematical background, equivalent to a 1-semester undergraduate course in each of the following: linear algebra, multivariate calculus (primarily differential calculus), probability theory, and statistics. (The coverage in DS-GA 1002 is sufficient.)
• Python programming required for most homework assignments.
• Recommended: Computer science background up to a "data structures and algorithms" course
• Recommended: At least one advanced, proof-based mathematics course
• Some prerequisites may be waived with permission of the instructor

Grading

Homework (35%) + One-Hour Test (15%) + Two-Hour Test (30%) + Project (20%)

Many homework assignments will have problems designated as “optional”. At the end of the semester, strong performance on these problems may lift the final course grade by up to half a letter grade (e.g. B+ to A- or A- to A), especially for borderline grades. You should view the optional problems primarily as a way to engage with more material, if you have the time. Along with the performance on optional problems, we will also consider significant contributions to Piazza and in-class discussions for boosting a borderline grade.

Important Dates

• First test (50 min) Wednesday, March 1st, 8:35–9:25pm.
• Second test (100 min) Tuesday, April 18th, 5:20–7pm.
• See Assignments section for homework-related deadlines.
• See Project section for project-related deadlines.

Textbooks

The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)

This will be our main textbook for L1 and L2 regularization, trees, bagging, random forests, and boosting. It's written by three statisticians who invented many of the techniques discussed. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)

An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)

This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)

Understanding Machine Learning: From Theory to Algorithms (Shalev-Shwartz and Ben-David)

Last year this was our primary reference for kernel methods and multiclass classification, and we may use it even more this year. Covers a lot of theory that we don't go into, but it would be a good supplemental resource for a more theoretical course, such as Mohri's Foundations of Machine Learning course. (Available for free as a PDF.)

Pattern Recognition and Machine Learning (Christopher Bishop)

Our primary reference for probabilistic methods, including bayesian regression, latent variable models, and the EM algorithm. It's highly recommended, but unfortunately not free online.

Bayesian Reasoning and Machine Learning (David Barber)

A very nice resource for our topics in probabilistic modeling, and a possible substitute for the Bishop book. Would serve as a good supplemental reference for a more advanced course in probabilistic modeling, such as DS-GA 1005: Inference and Representation (Available for free as a PDF.)

Other tutorials and references

• Carlos Fernandez-Granda's lecture notes provide a comprehensive review of the prerequisite material in linear algebra, probability, statistics, and optimization.
• Brian Dalessandro's iPython notebooks from DS-GA-1001: Intro to Data Science
• The Matrix Cookbook has lots of facts and identities about matrices and certain probability distributions.
• Stanford CS229: "Review of Probability Theory"
• Stanford CS229: "Linear Algebra Review and Reference"
• Math for Machine Learning by Hal Daumé III

Software

• NumPy is "the fundamental package for scientific computing with Python." Our homework assignments will use NumPy arrays extensively.
• scikit-learn is a comprehensive machine learning toolkit for Python. We won't use this for most of the homework assignments, since we'll be coding things from scratch. However, you may want to run the scikit-learn version of the algorithms to check tha your own outputs are correct. Most people will use it for their final projects. Also, studying the source code can be a good learning experience.

Overview

The project is your opportunity for in-depth engagement with a data science problem. In job interviews, it's often your course projects that you end up discussing, so it has some importance even beyond this class. That said, it's better to pick a project that you will be able to go deep with (in terms of trying different methods, feature engineering, error analysis, etc.), than choosing a very ambitious project that requires so much setup that you will only have time to try one or two approaches.

Key Dates

• Feb 28 (Tues 5pm): Deadline for choosing project groups
• March 6 (Mon 9pm): Email one-sentence project idea(s) to adviser, along with personal intros
• March 8 (Wed 8:35–9:25pm): First meeting with advisers. Each group will give a 5-minute presentation of their project idea to their assigned project adviser, followed by brief discussing with adviser.
• March 23 (Thurs 6pm): Project Proposals Due
• Apr 19th (Wed 8:35–9L25pm): Second meeting with advisers
• May 3rd (Wed 8:35–9:25pm): Third meeting with advisers
• May 9th (Tues 5–7pm): Project Poster Session in CDS, 7th floor
• May 12th, 6pm: Final Project Reports Due

Guidelines for Project Topics

A good project for this class is one that's a real "problem", in the sense that you have something you want to accomplish, and it's not necessarily clear from the beginning the best approach. The techiques used should be relevant to our class, so most likely you will be building a prediction system. A probabilistic model would also be acceptable, though we will not be covering these topics until later in the semester.

To be clear, the following approaches would be less than ideal:

1. Finding an interesting ML algorithm, implementing it, and seeing how it works on some data. This is not appropriate because I want your choice of methods to be driven by the problem you are trying to solve, and not the other way around.
2. Choosing a well-known problem (e.g. MNIST digit classification or the Netflix problem) and trying out some of our ML methods on it. This is better than the previous example, but with a very well-established dataset, a lot of the most important and challenging parts of real-world data science are left out, including defining the problem, defining the success metric, and finding the right way to encode the data.
3. Choosing a problem related to predicting stock prices. Historically, these projects are the most troubled. Interestingly, our project advisers who have worked in this field are the ones who advise against this most strongly.

Project proposal guidelines

The project proposal should be roughly 2 pages, though it can be longer if you want to include figures or sample data that will be helpful to your presentation. Your proposal should do the following:

1. Clearly explain the high-level problem you are trying to solve (e.g. predict movie ratings, predict the outcome of a court case, ...).
2. Identify the data set or data sets that you will be using. You should give a clear description of the characteristics of the data (how many examples, what kinds of features do we have for each example, are there issues with missing data or bad data, etc.).
3. How will you evaluate performance? In certain settings, you may want to try a few different performance measures.
4. Identify a few "baseline algorithms". These are simple algorithms for solving the problem, such as always predicting the majority class for a classification problem, using a small set of decision rules designed by hand, or using a ridge regression model on a basic feature set. Ideally, you will be able to report the performance of a couple baseline algorithms in your proposal. The goal will be to beat the baseline, so if the baseline is already quite high, you will have a challenge.
5. What methods do you plan to try to solve your problem, along with a rough timeline. Methods include data preprocessing, feature generation, and the ML models you'll be trying. Once you start your investigation, it's best to use an iterative approach, where the method you choose next is based on an understanding of the results of the previous step.

Some Previous Projects

• Social attitudes cannot be predicted from federal court decisions and judge characteristics.
• What Matters: Agreement between U.S. Courts of Appeals Judges
• Probability Estimation for Online Education System
• Using Predictive Models to Determine Judge Bias in Asylum Refugee Court Cases
• E-Commerce Recommender Modeling: A Case Study of Ponpare Coupon Purchase Prediction.
• Predicting Job Markets in India using heavy tailed regression
• Resampling the Spectral Time Series of a SN in the Time Dimension

Some Public Data Sets (just to get you thinking)

• Datasets on Amazon's AWS cloud
• Yelp Dataset Challenge
• NYC Open Data
• Data.gov
• UN Data
• Kaggle
• Quandl financial, economic, social datasets
• Rating data sets from MovieLens
• Congress voting records
• Quota's meta list of datasets

Instructor

David Rosenberg

dr129@nyu.edu

David is a data scientist in the office of the CTO at Bloomberg L.P. Formerly he was Chief Scientist of YP Mobile Labs at YP.

Teaching Assistants

Brett Bernstein

brettb@cims.nyu.edu

Brett is a third year PhD student in the Math department at Courant working with Prof. Carlos Fernandez-Granda

Vladimir Kobzar

vkobzar@cims.nyu.edu

Vlad is a math graduate student at Courant Institute, where he works on algorithms at the intersection of mathematics and machine learning. He is also a lawyer and was previously an Executive Director at Goldman Sachs.