This week

Prerequisites

• DS-GA-1001: Intro to Data Science or its equivalent
• DS-GA-1002: Statistical and Mathematical Methods or its equivalent
• Solid mathematical background, equivalent to a 1-semester undergraduate course in each of the following: linear algebra, multivariate calculus (primarily differential calculus), probability theory, and statistics. (The coverage in DS-GA 1002 is sufficient.)
• Python programming required for most homework assignments.
• Recommended: Computer science background up to a "data structures and algorithms" course
• Recommended: At least one advanced, proof-based mathematics course
• Some prerequisites may be waived with permission of the instructor.

Grading

Homework (40%) + One-Hour Test (15%) + Two-Hour Test (25%) + Project (20%)

Many homework assignments will have problems designated as “optional”. At the end of the semester, strong performance on these problems may lift the final course grade by up to half a letter grade (e.g. B+ to A- or A- to A), especially for borderline grades. You should view the optional problems primarily as a way to engage with more material, if you have the time. Along with the performance on optional problems, we will also consider significant contributions to Piazza and in-class discussions for boosting a borderline grade.

Important Dates

• First test (50 min) Thursday, March 3rd, 7:10-8pm.
• Second test (110 min) Wednesday, April 13th, 7:10-9pm.
• See Assignments section for homework-related deadlines.
• See Project section for project-related deadlines.

Textbooks

The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)

This will be our principal textbook for the first part of the course. It's written by three statisticians who invented many of the techniques discussed. Despite its popularity and the pretty pictures, this is not an easy book. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)

An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)

This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)

Bayesian Reasoning and Machine Learning (David Barber)

We'll use this as a reference for probabilistic modeling, including Bayesian methods, and Bayesian networks. (Available for free as a PDF.)

Pattern Recognition and Machine Learning (Christopher Bishop)

This book is another very nice reference for probabilistic models and beyond. It's highly recommended.

Machine Learning: A Probabilistic Perspective (Kevin P. Murphy)

This book covers an unusually broad set of topics, including recent advances in the field. As such, it's a great reference to have, particularly if you continue your study of data science beyond this course. That said, it was the required textbook for this course in 2015, and many students found it a bit overwhelming. It's really intended as a comprehensive, PhD-level textbook.

Convex Optimization (Boyd and Vandenberghe)

This book was an instant hit in the machine learning community when it was published in 2004. We will be making light use of this book, mostly for its coverage of Lagrangians and duality. However, it's a good book to get familiar with, as it's very well written, and it covers a lot of techniques used in more advanced machine learning literature. (Available for free as a PDF.)

Other tutorials and references

(If you find additional references that you recommend, please share them on Piazza and we can add them here.)

• Carlos Fernandez-Granda's lecture notes provide a comprehensive review of the prerequisite material in linear algebra, probability, statistics, and optimization.
• The Matrix Cookbook has lots of facts and identities about matrices and certain probability distributions.
• Stanford CS229: "Review of Probability Theory"
• Stanford CS229: "Linear Algebra Review and Reference"
• Math for Machine Learning by Hal Daumé III
• Brian Dalessandro's iPython notebooks from DS-GA 1001: Introduction to Data Science

Software

• NumPy is "the fundamental package for scientific computing with Python." Our homework assignments will use NumPy arrays extensively.
• scikit-learn is a comprehensive machine learning toolkit for Python. We won't use this for most of the homework assignments, since we'll be coding things from scratch. However, you may want to run the scikit-learn version of the algorithms to compare the outputs to your own, as a check. Besides that, it should be useful for many final projects, and studying the source code can be a good learning experience. One of the core developers, Andreas Müller, is a Research Engineer in NYU's Center for Data Science.
• CVXPY and CVXOPT are for solving convex optimization problems in Python. Could be useful for checking your homework results.

Overview

The project is your opportunity for in-depth engagement with a data science problem. In job interviews, it's often your course projects that you end up discussing, so it has some importance even beyond this class. That said, it's better to pick a project that you will be able to go deep with (in terms of trying different methods, feature engineering, error analysis, etc.), than choosing a very ambitious project that requires so much setup that you will only have time to try one or two approaches.

Key Dates

• Feb 29 (Mon 6pm): Deadline for choosing project groups
• March 10 (Thur 7-9pm): First meeting with advisers. Each group will give a 5-minute "pitch" of their project idea to their assigned project adviser. The adviser may give immediate feedback or ask follow-up questions.
• March 24 (Thurs 6pm): Project Proposals Due
• Apr 14th (Thurs 7-9pm): Second meeting with advisers
• May 5th (Thurs 7-9pm): Third meeting with advisers
• May 11th (Wed 6-8pm): Project Poster Session
• May 13th, 6pm: Final Project Reports Due

Guidelines for Project Topics

A good project for this class is one that's a real "problem", in the sense that you have something you want to accomplish, and it's not necessarily clear from the beginning the best approach. The techiques used should be relevant to our class, so most likely you will be building a prediction system. A probabilistic model would also be acceptable, though we will not be covering these topics until later in the semester.

To be clear, the following approaches would be less than ideal:

1. Finding an interesting ML algorithm, implementing it, and seeing how it works on some data. This is not appropriate because I want your choice of methods to be driven by the problem you are trying to solve, and not the other way around.
2. Choosing a well-known problem (e.g. MNIST digit classification or the Netflix problem) and trying out some of our ML methods on it. This is better than the previous example, but with a very well-established dataset, a lot of the most important and challenging parts of real-world data science are left out, including defining the problem, defining the success metric, and finding the right way to encode the data.
3. Choosing a problem related to predicting stock prices. Historically, these projects are the most troubled. Interestingly, our project advisers who have worked at hedge funds are the ones who advise against this most strongly.

Project proposal guidelines

The project proposal should be roughly 2 pages, though it can be longer if you want to include figures or sample data that will be helpful to your presentation. Your proposal should do the following:

1. Clearly explain the high-level problem you are trying to solve. (e.g. Predict movie ratings, predict the outcome of a court case, find a low-dimensional characterization of input examples.)
2. Identify the data set or data sets that you will be using. You should give a clear description of the characteristics of the data (how many examples, what kinds of features do we have for each example, are there issues with missing data or bad data, etc.).
3. How will you evaluate performance? In certain settings, you may want to try a few different performance measures.
4. Identify a few "baseline algorithms". These are simple algorithms for solving the problem, such as always predicting the majority class for a classification problem, using a small set of decision rules designed by hand, or using a ridge regression model on a basic feature set. Ideally, you will be able to report the performance of a couple baseline algorithms in your proposal, though this is not necessary. The goal will be to beat the baseline, so if the baseline is already quite high, you will have a challenge.
5. What methods do you plan to try to solve your problem, along with a rough timeline. Methods include data preprocessing, feature generation, and the ML models you'll be trying. Once you start your investigation, it's best to use an iterative approach, where the method you choose next is based on an understanding of the results of the previous step.

Some Public Data Sets (just to get you thinking)

• Datasets on Amazon's AWS cloud
• Yelp Dataset Challenge
• NYC Open Data
• Data.gov
• UN Data
• Kaggle
Quandl financial, economic, social datasets
• Face recognition, collaborative filtering, web ranking (see bottom, under "Projects")
• More collaborative filtering data
• 20 Newsgroups
• Blogs (with spam labels)
• Enron e-mail data set
• More Enron resources
• Congress voting records
• Quota's meta list of datasets

Instructor

David Rosenberg

dr129@nyu.edu

David is a data scientist in the office of the CTO at Bloomberg L.P. Formerly he was Chief Scientist of YP Mobile Labs at YP.

Teaching Assistant

Levent Sagun

sagun@cims.nyu.edu

Levent is a PhD student at the Courant Institute of Mathematical Sciences.