Agenda

Special Session on "Data Perspectivism in Ground Truthing and Artificial Intelligence" - S3

Description

Many Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process (often called ground-truthing) is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. Recently, a different paradigm for ground-truthing has started to emerge, called data perspectivism, which moves away from traditional majority aggregated datasets, towards the adoption of methods that integrate different opinions and perspectives within the knowledge representation, training, and evaluation steps of ML processes, by adopting a non-aggregation policy. This alternative paradigm obviously implies a radical change in how we develop and evaluate ML systems: such ML systems have to take into account multiple, uncertain, and potentially mutually conflicting views. 

The scope of this special session is to attract contributions related to the management of subjective, uncertain, multi-perspective, or otherwise non-aggregated data in ground-truthing, machine learning, and more generally artificial intelligence systems.

Invited contributions: full research papers and research in progress papers.

Topics of interest:

• Subjective, uncertain, or conflicting information in annotation and crowdsourcing processes;
• Limits and problems with standard data annotation and aggregation processes;
• Theoretical studies on the problem of learning from multi-rater and non-aggregated data;
• Participation mechanisms/incentives/gamification for rater engagement and crowdsourcing;
• Ethical and legal concerns related to annotation and aggregation processes in ground-truthing;
• Creation and documentation of multi-rater and non-aggregated datasets and benchmarks;
• Development of ML algorithms for multi-rater and non-aggregated data;
• Development of techniques to detect and manage multiple forms of uncertainty in multi-rater and non-aggregated data;
• Techniques for the evaluation of ML systems based on multi-rater and non-aggregated data;
• Applications of data perspectivism and non-aggregated data to interpretable, human-in-the-loop AI and algorithmic fairness;
• Experimental and application studies of ML/AI systems on multi-rater and non-aggregated data, in possibly different application domains

Important dates

• Paper Submission deadline: February 25, 2022
• Notification of acceptance: April 1st, 2022
• Camera ready due: April, 22nd, 2022
• IPMU Conference: July 11th -15th, 2022

Author Guidelines:
Please refer to the IPMU 2022 page where guidelines and templates are available, in the main conference Web site.
All submissions accepted for presentation at IPMU 2022 will be published in the Communications in Computer and Information Science (CCIS) series, by Springer.            

IPMU 2022 S3 Special Session Co-Chairs:

• Andrea Campagner (University of Milano-Bicocca, IT)
• Teresa Scantamburlo (Ca’ Foscari University of Venice, IT)
• Valerio Basile (University of Turin, IT)
• Federico Cabitza (University of Milano-Bicocca, IT