DEEP LEARNING FOR NATURAL LANGUAGE PROCESSING

The course introduces the theoretical, methodological and practical foundations of Natural Language Processing (NLP), tracing its development from classical statistical approaches to modern Large Language Models (LLMs).

The course follows a progressive structure. After a non-technical introduction to NLP and LLMs, students are introduced to the mathematical and machine-learning foundations required to understand contemporary language models. The course then covers text processing and tokenization, n-gram models, text classification, neural networks, distributed word representations and neural language models.

The central part of the course focuses on sequence-to-sequence models, attention mechanisms, the Transformer architecture and the main pretraining objectives. The final part covers language generation, fine-tuning, semantic retrieval, Retrieval-Augmented Generation (RAG), the evaluation of NLP systems and LLMs, parameter-efficient fine-tuning, long-context models, post-training, alignment, and data-centric and responsible NLP.

Theoretical lectures are complemented by practical Python activities involving PyTorch, the Transformers library and the implementation of a decoder-only Transformer model.

The course aims to provide students with a broad and structured understanding of modern methods for the automatic analysis and generation of language. It also develops the ability to select, implement, train and evaluate models appropriate to specific practical problems.

By the end of the course, students will be able to:

- understand the linguistic, mathematical and computational foundations of NLP;
- describe the development from statistical language models to neural models and Transformers;
- process and tokenize text collections;
- implement and apply algorithms for text classification, representation and generation;
- understand and apply neural networks, recurrent models, sequence-to-sequence models and attention mechanisms;
- understand the Transformer architecture and the main families of language models;
- use PyTorch and Transformers to develop and train NLP models;
- apply full fine-tuning and parameter-efficient fine-tuning techniques;
- build systems for sentence representation, semantic retrieval and RAG;
- select appropriate metrics and protocols for evaluating NLP systems and LLMs;
- understand the principles of post-training, alignment, scaling and long-context modelling;
- critically assess data quality, provenance and representativeness, as well as the risks, limitations and social impact of NLP systems;
- select models and methodologies appropriate to the requirements of a specific application.

Basic knowledge of linear algebra, probability, statistics and machine learning is recommended. Two lectures will be used to introduce relevant concepts.

Basic Python programming skills are required for the practical activities. Previous familiarity with PyTorch and the Transformers library is useful but not mandatory.

Type Title
Intro Introduction to NLP and LLMs
Primer Mathematical and Learning Foundations
Lecture Text Processing and Tokenization
Lecture N-gram Language Models
Lecture Logistic Regression and Text Classification
Primer Neural Networks and Backpropagation
Lecture Distributed Word Representations
Lab PyTorch NLP Lab
Lecture Neural Language Models and RNNs
Lecture Seq2Seq and Classical Attention
Lecture Self-Attention
Lecture Transformer Architecture
Lecture Pretraining Objectives and Model Families
Lecture Language Generation
Lecture Structured and Text-to-Text NLP (fine-tuning)
Lab Transformers Coding Lab
Lecture Sentence Representations and Retrieval
Lecture Retrieval-Augmented Generation
Lecture Evaluation of NLP Systems and LLMs
Lecture Transformer Parameters and Parameter Efficient Fine-Tuning
Lecture Scaling and Long-Context Models
Lab Building a Decoder-Only Transformer Model Like Llama
Lecture Post-Training and Alignment
Lecture Data-Centric and Responsible NLP

Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd ed. draft), https://web.stanford.edu/~jurafsky/slp3/

Lecture slides, research papers, notebooks, code examples and all other required study materials will be made available through Moodle.

Any additional textbooks or resources will be indicated during the course.

Learning will be assessed through an oral examination covering the entire course syllabus. Assessment will consider knowledge of the topics, the ability to connect and apply concepts, critical thinking and appropriate use of technical terminology.

Students may optionally undertake an individual Python project agreed upon in advance with the instructor. The project will be assessed according to its methodological and technical quality, analysis of results and presentation, and may contribute up to 3 points to the final grade. The project does not replace the oral examination, and points will be awarded only if the examination is passed.

The examination is oral and covers the entire course syllabus. Students must demonstrate knowledge of the theoretical concepts, models, algorithms and methodologies addressed during the course.

Any optional individual project will be presented and discussed during the examination and may contribute up to 3 additional points, subject to the maximum grade permitted.

The instructor is responsible for ensuring the authenticity and originality of the assessment. In the event of suspected irregularities, an additional assessment may be required, including through a different format.

oral

The instructor is responsible for ensuring the authenticity and originality of all examinations and coursework. In cases of suspected academic misconduct, an additional on-site assessment may be required during the exams, which may differ from the standard format.

The evaluation criteria are as follows:

18–22: sufficient knowledge, basic application skills and adequate technical language.
23–26: fair knowledge, good ability to connect and apply concepts, and adequate critical thinking.
27–30: in-depth knowledge, autonomous application of concepts, and excellent critical and presentation skills.

Honours: excellent knowledge, full autonomy of judgement and the ability to produce original insights.

The course combines:

- lectures covering theoretical and methodological foundations;
- discussion of examples, case studies and research papers;
- guided exercises;
- Python programming laboratories;
- activities involving the design, implementation and evaluation of NLP systems.

The practical activities will allow students to consolidate the theoretical concepts through the use of PyTorch and Transformers and through the implementation of fundamental components of modern language models.