LEARNING WITH MASSIVE DATA

The goal of this course is to teach students to design and develop efficient and scalable algorithms for the analysis of large-scale data sources in distributed (cluster) environments.
The course investigates the aspect of algorithm scalability and software performance, experimenting differente development environments, and discussing parallel algorithms for data analysis and machine learning.
Some use cases are chosen among the topics of data mining, web search, and graph mining.

The course presents the fundamental techniques usually employed to solve large-scale data analysis problems with distributed algorithms.
Students acquire knowledge on models of parallel computing architectures, paradigms and environments of parallel programming, and design of algorithms for massive datasets.

Students will achieve the following learning outcomes:

i) Knowledge and understanding: understanding principles of distributed computing; understanding sources and models of costs massive datasets analysis solutions; understanding design patterns for massive data analysis.

ii) Applying knowledge and understanding: being able to design and develop algorithms for massive dataset analysis; being able to measure performance of a distributed program; being able to develop algorithms for massive dataset analysis by exploiting distributed programming patterns.

iii) Making judgements: being able to analyze different methods and algorithms and to choose the most appropriate to a given problem on the basis of a sound cost model.

iv) Communication skills: reporting a sound and comprehensive comparative analysis among different solutions supported by experiments.

v) Learning skills: being able to autonomously adopt new techniques and methods.

Students are expected to have a good background in algorithms and complexity, operating systems and Python programming.

- Recommender systems
- Classification, Clustering and Ranking
- Approximate Nearest Neighbour Search
- Large-scale parallelism, MapReduce, Apache Spark
- Association Rules Mining
- Graph Analysis

Lecture notes.

Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman. Mining of Massive Datasets 3rd Edition. Cambridge University Press 2020.

Learning outcomes are verified by a written exam and a the discussion of a some project assignments.

The written exam consists in questions and short exercises regarding the theory of the subjects discussed during the course. The written exam evaluates the achievement of the learning outcomes i), ii) e iii).

Each assignment requires to design and develop an algorithm for a given massive data analysis task. The student is asked to choose the most appropriate solution, to motivate its choice and to provide a report to be discussed with the teacher. The assignments evaluate the achievement of the learning outcomes iii) iv) e v).

The grade is given by 70% written exam plus 30% final project.

written and oral

The instructor is responsible for ensuring the authenticity and originality of all examinations and coursework. In cases of suspected academic misconduct, an additional on-site assessment may be required during the exams, which may differ from the standard format.

The grade is given by 70% written exam plus 30% final project.

28-30L: excellent mastery of the methods. Strong analytical and evaluation skills. Ability to adopt new software tools.
23-27: good mastery of the topics discussed in class, good clarity in presentation, and a solid ability to apply concepts to case studies.
18-22: basic knowledge of the topics discussed in class, limited proficiency in terminology, and an adequate ability to apply concepts to case studies.

Lectures and case studies.