Machine learning basics. This part briefly introduces the fundamental ML problems-- regression, classification, dimensionality reduction, and clustering-- and the traditional ML models and numerical algorithms for solving the problems.

• ML basics [slides-1] [slides-2].

• Regression [slides-1] [slides-2].

• Classification.

  • Logistic regression [slides] [lecture note].

  • SVM [slides].

  • Softmax classifier [slides].

  • KNN classifier [slides].

• Regularizations [slides-1] [slides-2].

• Clustering [slides].

• Dimensionality reduction [slides-1] [slides-2] [lecture note].

• Scientific computing libraries. [slides].

Neural network basics. This part covers the multilayer perceptron, backpropagation, and deep learning libraries, with focus on Keras.

• Multilayer perceptron and backpropagation [slides] [lecture note].

• Keras [slides].

• Further reading:

  • [activation functions]

  • [parameter initialization]

  • [optimization algorithms]

Convolutional neural networks (CNNs). This part is focused on CNNs and its application to computer vision problems.

• CNN basics [slides].

• Tricks for improving test accuracy [slides].

• Feature scaling and batch normalization [slides].

• Advanced topics on CNNs [slides].

• Popular CNN architectures [slides].

• Further reading:

  • [style transfer (Section 8.1, Chollet's book)]

  • [visualize CNN (Section 5.4, Chollet's book)]

Recurrent neural networks (RNNs). This part introduces RNNs and its applications in natural language processing (NLP).

• Categorical feature processing [slides] [video (Chinese)].

• Text processing and word embedding [slides] [video (Chinese)].

• RNN basics [slides] [video (Chinese)].

• LSTM [slides] [reference] [video (Chinese)].

• Making RNNs more effective [slides] [video (Chinese)].

• Text generation [slides] [video (Chinese)].

• Machine translation [slides] [video (Chinese)].

• Attention [slides] [video (English)] [video (Chinese)] [reference].

• Self-attention [slides] [video (English)] [video (Chinese)].

• Image caption generation [slides] [reference].

Transformer Models.

• Transformer (1/2): attention without RNN [slides] [video (English)] [video (Chinese)].

• Transformer (2/2): from shallow to deep [slides] [video (English)] [video (Chinese)] [reference].

• BERT: pre-training Transformer [slides] [video (English)] [video (Chinese)] [reference].

• Vision Transformer (ViT) [slides] [video (English)] [video (Chinese)].

Autoencoders. This part introduces autoencoders for dimensionality reduction and image generation.

• Autoencoder for dimensionality reduction [slides].

• Variational Autoencoders (VAEs) for image generation [slides].

Generative Adversarial Networks (GANs).

• DC-GAN [slides].

Deep Reinforcement Learning.

• Reinforcement learning basics [slides] [lecture note] [video (Chinese)].

• Value-based learning [slides] [video (Chinese)].

• Policy-based learning [slides] [video (Chinese)].

• Actor-critic methods [slides] [video (Chinese)].

• AlphaGo and Monte Carlo tree search [slides] [video (Chinese)].

Parallel Computing.

• Basics and MapReduce [slides] [lecture note] [video (Chinese)].

• Parameter server and decentralized network [slides] [video (Chinese)].

• TensorFlow's mirrored strategy and ring all-reduce [slides] [video (Chinese)].

• Federated learning [slides] [video (Chinese)].

Adversarial Robustness. This part introduces how to attack neural networks using adversarial examples and how to defend from the attack.

• Data evasion attack and defense [slides] [lecture note].

• Data poisoning attack [slides] [video (Chinese)].

• Further reading: [Adversarial Robustness - Theory and Practice].

Meta Learning.

• Few-shot learning: basic concepts [slides] [video (English)] [video (Chinese)].

• Siamese network [slides] [video (English)] [video (Chinese)].

• Pretraining + fine tuning [slides] [video (English)] [video (Chinese)].

Neural Architecture Search (NAS).

• Basics [slides] [video (Chinese)].

• RNN + Reinforcement Learning: [slides] [video (Chinese)].

• Differentiable NAS: [slides] [video (Chinese)].