Program

State-of-the-art lab and corpus methods, comprehensive theory, tailor-made statistics and computing, professional skills

An advanced program uniting corpus-based research, theoretical linguistics, and specialized methods in statistics and computing

The MULTICOM master’s program is 2 years, 4 terms, 120 ECTS, with one compulsory professional project in term 3 and the possibility of a second professional placement at a suitable partner during term 4. A professional fair takes place at the end of term 2, and a student-organized conference during term 4. Each term students must earn 30 ECTS.

The MULTICOM experience

TERM 1. Introduction to multimodal data science

September-December. University of Murcia, Spain

Welcome week, inauguration, social program
Thorough overview of multimodal data science
Intensive training in basic or intermediate quantitative skills
Basic professional skills
Opportunities to pursue key topics in the field, participate in hands-on research, fill in any gaps on language and cognition, or develop further professional and entrepreneurial skills.

TERM 2. The multimodality lab: cognition and learning

January-June. Lund University, Sweden

Welcome week, social program
Specialized training in laboratory-based methods and on theoretical background for research on multimodality in language, cognition, and learning
Intermediate or advanced training in quantitative skills
Data management, property rights, and ethics
Opportunity to gain further expertise in hands-on experimental research projects, to delve into the connections between multimodality and natural and artificial intelligence, or pursue a key topic in the field
Professional fair

TERM 3. Machine learning and multimodal corpora

September-January. FAU, Erlangen Campus, Germany

Professional project (internship) of at least 6 weeks between June and October
Specialized training in multimodal corpus linguistics, multimodal language models, and the creation, curation, analysis, and management of large-scale audiovisual data sets
Advanced professional skills
Opportunities to further master key technologies: machine-learning techniques, computer vision, speech analysis, and natural language understanding, or to pursue key topics in the field
Conference at the end of term, co-organized by students and faculty

TERM 4. Research and professional development

January-August. Location(s) of choice

Work on the MA thesis at one or two organizations, beneficiaries or associated partners, chosen by the student and the main supervisor, who must be from one of the three universities. Stays at non-academic partners, as well as professional training related to the thesis research, will be encouraged
Further professional training for both academic and industry careers
Thesis defense (virtual)
Graduation ceremony

How to Apply

Do you have more questions?

Contact us

courses

Term 1. University of Murcia

Core modules (21 ECTS)

Human Multimodal Communication (6 ECTS)

This course will offer a general introduction to the field of multimodal communication. The aim is to equip students with the necessary basic background knowledge of the field to multimodality by offering a general perspective to the main theories and applications in research. The students will acquire the required knowledge to undertake more specialised courses at later stages in the MA devoted to the study of specific modalities, the application of some of the theories in research, the creation of new tools in multimodal communication and real-world applications of multimodal research. In this way, the course favours a broad, panoramic perspective into the field and the integration of modalities such as language, gesture, image, or prosody, rather than a more detailed look at each of these modes of communication. The object of the course will be multimodal communication in general. Combinations of patterns from various modalities and examples from a wide array of world languages will be used.

Methods in multimodal data science (6 ECTS)

This course will offer a general introduction to the main tools and methods that are currently used in the data-based study of human multimodal communication. The aim of this course is to provide students with the basic skills to engage with a wide range of tools and software that will be vital in later specialised courses. Students will be introduced to the main types of methods: experimental in lab settings, corpus based on video and audio datasets, and fieldwork. They will acquire basic transversal skills ranging from command-line familiarity for programming (basic Linux shell commands), experimental design, data management, (data structure and formats), ethics in data management, and specialised software for research into multimodal communication (ELAN, MULTIDATA, Praat, Psychopy, among others). The course will combine both the formal description and the basic guidelines of the different methods and tools together with their application in practical cases. The course will favour a general, introductory approach to these methods, preparatory for later courses).

Professional impact I (3 ECTS)

This course will present students with the main strategies and tools for developing a strong professional impact in academic and professional contexts. The aim is to equip students with the skills needed to communicate effectively, present their work confidently, and build a professional presence. Students will acquire foundational knowledge in areas such as academic and professional communication (e.g., structuring a clear argument, tailoring communication to different audiences), presentation skills (e.g., design and delivery of effective presentations), networking strategies (e.g., establishing professional connections,), self-presentation (e.g., CV writing, personal statements, professional portfolios) and writing (e.g. grant writing guide) skills. They will also learn how to reflect critically on their own professional goals and to apply strategies that enhance their visibility and credibility in their chosen field. The course will consist of interactive sessions combining theoretical insights with practical exercises, enabling students to practise and apply the tools introduced.

Statistics and computing (6 ECTS) OR Advanced statistics and computing (core module, 6 ECTS)

Students will be placed in one of the two options, according to their level:

Statistics and computing. This course will provide an introduction to the main statistical methods used in the analysis of experimental and numerical data. The aim is to equip students with the basic skills needed to describe and interpret data through a variety of approaches. Students will acquire foundational knowledge in measures of central tendency and dispersion, data visualisation techniques, treatment of extreme values, data distribution, and data transformation. They will also learn to distinguish between different types of variables (e.g., categorical, ratio scale, proportions, frequencies), to apply the principles of hypothesis testing, and to create well-structured data files (e.g., using spreadsheet software). In addition, students will practise summarising data sets with appropriate descriptive statistics and using the software R to produce diagrams and conduct statistical analyses. No previous knowledge of R will be required: only the basic knowledge of statistics and data-based methods mandatory for applying to MULTICOM.

Advanced statistics and computing. This course will introduce students to the principles and practices of statistical analysis through a combination of lectures and practical exercises. The aim is to develop the skills needed to prepare, analyse, and report data in a clear and scientific manner. Students will learn to apply the necessary modifications to datasets, to select and calculate suitable descriptive statistics, and to choose statistical models that either provide an adequate fit or address the research questions under investigation. They will also acquire the ability to follow up initial models with additional analyses when required and to present their findings in a precise and well-structured style.

Language and cognition (elective course, 6 ECTS)

This is an intensive course of guided, self-paced study, geared towards students who have been waived the statistics & computing courses but need to fill-in gaps in areas such as semantics, pragmatics, phonetics, general linguistics, or the cognitive sciences related to language. Students taking this course will be pre-assessed to identify their individual needs to comply with the advanced background in language, communication, and thought required at the end of term 1. Individual needs will be addressed through both guided study and joint activities. The course will mainly take the form of an active seminar based on reading assignments, with oral presentations and direct interaction between students and faculty. There will also be tutoring sessions in small numbers and a limited number of lectures.

Elective courses (9 ECTS)

Language and Cognition (6 ECTS)

Studying the multimodal flow of language (6 ECTS)

A hands-on research course on real case-studies inspired in recent or ongoing funded research projects at UMU, with the opportunity of participating in activities leading to publishable research results, and to collaborate in ongoing research projects. Students will collaborate closely with experienced researchers, learning to build and curate video datasets for the study of multimodal behaviors in relation with specific linguistic patterns (phonetic variation, grammatical constructions, syntactic structures, semantic distinctions, pragmatic functions). Through user-friendly interfaces such as the MULTIDATA platform (https://multi-data.eu/), students will analyze data with various computational tools and statistical and machine-learning techniques. No previous technical, statistical, or methodological knowledge will be required, beyond what will be taught simultaneously in the methods and statistics courses. An introduction to basic tenets and research questions will be followed by hands-on demonstrations for data gathering, as well as by data analysis sessions and joint discussions of results and their relevance.

Management and business (6 ECTS)

Crash course on managerial skills: team building, personal communication, social entrepreneurship, industry-specialized conferences, financial reporting and analysis, economic environment and country economic analysis, management accounting, leadership of people and change, strategic marketing, negotiation, creativity and ideation, influence and persuasion, government and sustainability, entrepreneurial venturing including start-up and spin-off development, creating value, digital transformation, leadership and uncertainty, or career planning with a management and business-administration component. This course is offered in terms 1 and 3, which allows both first- and second-year students to take it.

Trends in multimodal data science I (3 ECTS)

Online. University of Murcia Faculty. This course will introduce current trends and emerging approaches in multimodal data science. The aim is to familiarise students with recent developments in methods, tools, and applications, and to provide a framework for critically evaluating new research. The specific area of study that will be covered in the course will change according to the availability of the invited teaching staff and their expertise. Topics include (but are not limited to) the study of pioneering advanced data collection techniques (e.g., multimodal sensors, motion caption devices, virtual reality), innovative data analysis methods (e.g., machine learning for multimodal data, automated annotation), application of multimodal research in education (e.g. multimodality in developmental psychology). The course will combine theoretical discussions of emerging trends with practical examples and case studies, enabling students to apply these approaches to real-world multimodal datasets. This course is offered in terms 1 and 3, which allows both first- and second-year students to take it.

Trends in multimodal data science III (3 ECTS)

Online. FAU faculty. This course will introduce current trends and emerging approaches in multimodal data science. The aim is to familiarise students with recent developments in methods, tools, and applications, and to provide a framework for critically evaluating new research. The specific area of study that will be covered in the course will change according to the availability of the invited teaching staff and their expertise. Topics include (but are not limited to) the study of pioneering advanced data collection techniques (e.g., multimodal sensors, motion caption devices, virtual reality), innovative data analysis methods (e.g., machine learning for multimodal data, automated annotation), application of multimodal research in education (e.g. multimodality in developmental psychology). The course will combine theoretical discussions of emerging trends with practical examples and case studies, enabling students to apply these approaches to real-world multimodal datasets. This course is offered in terms 1 and 3, which allows both first- and second-year students to take it.

Free-choice course at the University of Murcia, outside the MULTICOM offer (6 ECTS max.)

Students are expected to obtain all necessary credits from the master’s own offer. However, individual demands for courses locally offered at UMU will be considered on a case-by-case basis. Such courses should be of clear theoretical or methodological relevance, and well integrated into the student’s degree and career plans. They will not be language courses. All students will be encouraged to take an extra course in Spanish as a foreign language from the UMU Language Service, as an extracurricular activity, not for credit, with tuition covered for MULTICOM-funded students.

Term 2. University of Lund

Core modules (20 ECTS)

Lab methods in multimodal data science (7.5 ECTS)

This course builds on and expands the course Human multimodal communication in semester 1, and offers a general introduction to a range of lab-based methods for the study of multimodal communication. It introduces an introduction to novel approaches to audiovisual recordings (e.g. 360-cameras) and to sensor-based technology such as eye-tracking, motion capture, articulography and biopac recordings. Students will be introduced to the theoretical foundations of each technique alternating with hands-on experience with the tools in smaller groups. The aim of the course is to familiarise students with theoretical and practical aspects of lab approaches for the study of human multimodal communication. It will provide them with basic practical skills in recording data from sensor equipment, and introduce them to software needed for the analysis of the recorded data. The course will provide formal descriptions and basic guidelines for the use of different tools together and exemplify their application in practical cases.

Language and multimodality (5 ECTS)

This course specialises on the tight relationship between multimodality and language use, whether spoken or signed. It introduces and discusses multimodality in relation to different linguistic levels (e.g., prosody, semantics, syntax, etc.), effects of crosslinguistic variation, (child and adult) language acquisition and bilingualism. It also discusses the relationship between gestures and sign language. The course will mainly take the form of an active seminar based on reading assignments, with oral presentations and direct interaction between students and faculty.

Ethics, data management, intellectual property (2.5 ECTS)

This course introduces and discusses the complexities of working on multimodal data (sound, video, revealing individuals), and the particular challenges related to what to do and not to do with such. The course discusses research ethics (e.g., what is required by law vs. what is morally just), challenges of data management (e.g., storage on secure servers to shield individuals’ identity and integrity vs. cloud storage), and the tension between issues of intellectual property and the move towards Open science and Open data. The course takes the shape of an active seminar based on readings and concrete examples, and direct interaction between students and faculty.

Introduction to programming in Python or Advanced statistics and computing (5 ECTS)

The course gives an introduction to Python programming as a tool to study human behaviour. It consists of lectures and practical exercises. The course gives an introduction to basic concepts in Python. (e.g. Numpy arrays, defining functions, basic graphics and plotting, stimulus preparation, etc.). It also provides practical training and students will be asked to define and solve a problem of their own using Python. No previous knowledge of Python is assumed.

Elective courses (15 ECTS)

Trends in multimodal data science II (5 ECTS)

This course will introduce current trends and emerging approaches in multimodal data science. The aim is to familiarise students with recent developments in methods, tools, and applications, and to provide a framework for critically evaluating new research. The specific area of study that will be covered in the course will change according to the availability of the invited teaching staff and their expertise. Topics include (but are not limited to) the study of advanced data collection techniques (e.g., multimodal sensors, motion capture devices, virtual reality), new data analyses (e.g., combination of data streams such as eye-tracking and motion capture). The course will combine theoretical discussions of emerging trends with practical examples and case studies.

Artificial cognitive systems and multimodality (5 ECTS)

This course will introduce current approaches to artificial cognitive systems and social robotics, focusing especially on approaches to making robots reproduce or respond to human multimodal behaviour (e.g. gaze direction for goal-directed action, emotional responses such as pupil dilation, gesture recognition and synthesis). The course will introduce theoretical and practical issues through seminar style lectures as well as practical experience with individual cases.

Lab project in multimodality (5 ECTS)

This course is intended to develop deeper knowledge and expertise concerning a lab-based approach to multimodality. The aim is to give student expanded insights into and practical knowledge of the steps in a multimodal project using specific equipment ranging from planning, data collection, data treatment and curation, visualisation, and analysis, and to train them in critically evaluating new research. The specific area of study and tools used in the course will change according to the availability of technology, invited teaching staff and their expertise. Students will work on developing a concrete project of their own.

Term 3. FAU Erlangen-Nürnberg

Core modules (22.5 ECTS)

Professional project (associated partner) (5 ECTS)

In their Professional Project at an associated partner (universities, industry, public administration, non-profit organizations), students apply their knowledge and skills in a professional setting. Typical duration: 6 weeks full-time.

Applications of machine learning (10 ECTS)

This module provides a hands-on introduction to the application of machine learning in multimodal communication research. Students will explore key areas including Natural Language Processing, speech processing, computer vision, and their combinations. The module emphasizes practical experience, with projects tailored to individual programming skills and research interests. Students may engage with high-level applications or implement low-level technical solutions, gaining both conceptual understanding and practical expertise in applying machine learning techniques to real-world multimodal data.

Advanced seminar (Hauptseminar):
In this seminar, students will gain practical experience in applying machine learning to multimodal communication research. They will learn to use off-the-shelf machine learning tools as well as develop their own custom pipelines. Students will design and carry out their own projects, demonstrating skills in building and annotating multimodal corpora from scratch. The seminar emphasizes hands-on learning, enabling students to integrate theory and practice while developing expertise in managing and analyzing complex multimodal data.
Practice seminar (Übungsseminar):
In this practice seminar, students will learn to work with large-scale datasets using high-performance computing (HPC) resources. They will gain hands-on experience in scheduling jobs, monitoring performance, and optimizing computational workflows (e.g., parallel computing). Additionally, students will explore best practices for data management, reproducibility, and efficient resource utilization, preparing them to handle complex machine learning tasks on large multimodal datasets covered in the advanced seminar.

Multimodal Corpus Linguistics (5 ECTS)

This module focuses on the design, creation, and use of multimodal corpus resources. Students will learn how to combine various data streams, including the output gathered from the various machine learning applications presented in the Applications module.

Seminar:
In this seminar, students will learn about language corpora, starting with pure text corpora, moving on to spoken and multimodal corpora. Students will learn how to collect their own multimodal corpora, how to structure metadata and annotation using various file formats, and how to exploit such corpora for linguistic research.

Professional impact II (2.5 ECTS)

Advanced communication skills, organizing and writing the master’s thesis, peer-review and publication, speaking to multidisciplinary audiences, visual and oral (or multisensory) presentation of data, advanced project writing and management, job applications and interviews, grant/fellowship writing, business and academic CV writing and presentation. Additionally, individualized networking strategies as well as team work to finalize the organization of the MULTICOM conference.

Elective courses (17.5 ECTS)

Management and business (5 ECTS)

Trends in multimodal data science I (2.5 ECTS)

Trends in multimodal data science III (2.5 ECTS)

This course will introduce current trends and emerging approaches in multimodal data science. The aim is to familiarise students with recent developments in methods, tools, and applications, and to provide a framework for critically evaluating new research. The specific area of study that will be covered in the course will change according to the availability of the invited teaching staff and their expertise. Topics include (but are not limited to) the study of pioneering advanced data collection techniques (e.g., multimodal sensors, motion caption devices, virtual reality), innovative data analysis methods (e.g., machine learning for multimodal data, automated annotation), application of multimodal research in education (e.g. multimodality in developmental psychology). The course will combine theoretical discussions of emerging trends with practical examples and case studies, enabling students to apply these approaches to real-world multimodal datasets. This course is offered in terms 1 and 3, which allows both first- and second-year students to take it.

Work project in multimodal data science (2.5 ECTS)

Individual professional project at a non-academic partner, co-supervised by a faculty member from LU/FAU/UMU and a tutor at the partner. Flexibility in content and schedule, upon approval of the graduate advisor and governing board.

Possibilities:

follow-up from the professional project
a small project in anticipation of a further placement in term 4
an exploratory application of the student’s skills to a non-academic organization
an open-ended collaboration in research in technology development, in UX or CX, or any other relevant field.

Multimodal Computer Science (5 ECTS)

This module allows students to take relevant courses from Computer Science, e.g.

Machine Learning for Engineers I (Eskofier)
Introduction to Machine Learning (Christlein)
Deep Learning (Maier)

Other modules or courses from Computer Science, Data Science, Digital Humanities and Computational Linguistics might be imported if approved by the module coordinator.