The Revival
1990s-2010s · Rise of Deep Learning
In the mid-1990s, with improved computing power and the internet's proliferation, machine learning, especially deep learning, began to emerge. This period in AI development can be described as the "statistical revolution"—shifting from rule-based systems to probabilistic and data-driven models. In 2012, deep learning won the ImageNet competition by a landslide, marking the beginning of the deep learning revolution. Since then, AI has achieved unprecedented breakthroughs in image recognition, speech recognition, and natural language processing. In 2016, AlphaGo's victory over Lee Sedol brought AI into the public consciousness, opening a new era of artificial intelligence. This era is characterized by the combination of big data, powerful computing resources, and deep learning algorithms. The proliferation of the internet made it possible to collect massive amounts of data, while GPU development provided the computational foundation for training large neural networks.
Key Milestones
Rise of Statistical Methods
In the 1990s, statistical methods began to dominate AI research. Hidden Markov Models (HMMs) in speech recognition, Naive Bayes and decision trees in machine learning, all achieved good results.
Birth of the World Wide Web
Tim Berners-Lee created the World Wide Web. The internet's proliferation changed human lifestyle and provided unprecedented data resources for AI.
Support Vector Machine
Vladimir Vapnik et al. proposed Support Vector Machine (SVM), a powerful classification and regression method. SVM achieved excellent performance in text classification and handwritten recognition.
Vladimir Vapnik
Long Short-Term Memory (LSTM)
Sepp Hochreiter and Jürgen Schmidhuber proposed LSTM, a special type of RNN capable of learning long-term dependencies. LSTM achieved great success in speech recognition and language modeling.
Sepp Hochreiter & Jürgen Schmidhuber
RNN for Language Modeling
Yoshua Bengio proposed using RNNs for language modeling. This work laid the foundation for later large-scale language models.
Yoshua Bengio
LeNet-5
Yann LeCun developed LeNet-5, a CNN for handwritten digit recognition. LeNet-5 was the first successfully commercialized deep learning model, used in the US Postal Service for ZIP code recognition.
Yann LeCun
NIPS Deep Learning Workshop
Geoffrey Hinton organized a deep learning workshop at NIPS. This conference is considered the starting point of the deep learning revival.
Geoffrey Hinton
Deep Belief Networks
Geoffrey Hinton proposed Deep Belief Networks (DBN), the first successfully trained deep neural network. Hinton introduced "layer-wise pre-training" to solve training difficulties.
Geoffrey Hinton
GPU for Deep Learning
Researchers began using GPUs to accelerate deep learning training. GPU's parallel computing made training large neural networks possible.
ImageNet Dataset
Fei-Fei Li's team created the ImageNet dataset, containing over 14 million annotated images. An unprecedented large-scale image dataset for computer vision research.
Fei-Fei Li
Speech Recognition Breakthrough
Google used deep learning models to reduce speech recognition error rate by 25%. This was the first major breakthrough of deep learning in speech recognition.
ImageNet Breakthrough
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton's team used AlexNet in the ImageNet competition, winning by a landslide. Error rate dropped from 26% to 15%.
AlexNet Team
Word2Vec
Tomas Mikolov released Word2Vec, a neural network model for learning word embeddings. Word2Vec enabled computers to understand semantic relationships between words.
Tomas Mikolov
Generative Adversarial Networks (GAN)
Ian Goodfellow proposed GAN, one of the most important innovations in deep learning. GAN generates extremely realistic images through adversarial training.
Ian Goodfellow
ResNet
Kaiming He et al. proposed ResNet, solving the training difficulty of deep networks through skip connections. ResNet enabled training hundreds of layers.
Kaiming He
AlphaGo Defeats Lee Sedol
DeepMind's AlphaGo defeated world Go champion Lee Sedol 4-1. Go was considered one of the most difficult board games for AI.
DeepMind AlphaGo
Transformer Architecture
Google proposed the Transformer architecture, a revolutionary breakthrough in NLP. Transformer is the foundation of modern large language models.
Google Brain
Key Figures
Geoffrey Hinton
Father of Deep Learning
Hinton, along with Yann LeCun and Yoshua Bengio, received the Turing Award in 2018, known as the "Deep Learning Trio". His persistence and innovation have completely transformed AI.
Yann LeCun
Father of CNNs
LeCun is Chief AI Scientist at Facebook. His LeNet-5 was the first successfully commercialized deep learning model. His contributions to CNNs laid the foundation for modern computer vision.
Yoshua Bengio
Deep Learning Pioneer
Bengio is a professor at the University of Montreal and an important figure in deep learning. His contributions to RNNs and NLP are equally important.
Ian Goodfellow
Father of GAN
Goodfellow proposed GAN in 2014, a revolutionary method in generative AI. From deepfakes to artistic creation, GAN applications are ubiquitous.
Fei-Fei Li
Mother of ImageNet
Li is a professor at Stanford University. Her ImageNet dataset had profound impact on computer vision. She also co-founded AI4ALL, promoting diversity in AI education.
Classic Quotes
"Deep learning will change everything because it allows us to learn hierarchical representations from data."
— Geoffrey Hinton
"Deep learning works because it can automatically learn hierarchical representations of data."
— Yann LeCun
"Our goal is artificial general intelligence—building machines that learn and think like humans."
— Demis Hassabis
"No data, no intelligence. But more importantly, no human intelligence, no true intelligence."
— Fei-Fei Li
"The power of GAN is that it allows machines to "imagine"."
— Ian Goodfellow