Master thesis opportunity in multi-modal AI for social understanding

Facing the Challenges of our time

Help us grow and be more impactful!

The “AI & Vision Systems” group of our “Integrated & Wireless Systems” Business Unit based in Neuchâtel, Switzerland, is currently looking for a Master thesis in multi-modal AI for social understanding.

 

Your Mission :

We are seeking a highly motivated master's student to advance our work in developing smarter social robots or human computer interaction (HCI), with a specific focus on leveraging data-driven approaches in social interaction. This master thesis will contribute to the development and enhancement of a multimodal deep learning system designed to classify social interactions from an egocentric perspective using video and audio data from the Ego4D dataset.

Your Responsabilities :

The primary responsability, over a minimum of 6 months, is to employ various monomodal and multimodal fusion techniques to improve the performance of machine learning models in accurately detecting "Talking to me" (TTM) and "Looking at me" (LAM) interactions. The objective is to:

• Investigate the problem and literature;
• recover the benchmark data and validate an experimental protocol;
• implement one baseline;
• research and explore multiple frameworks e.g. related to data fusion or problem formulations to improve the results.

In short, your objective is a classification problem that, based on two input modalities (video and audio), aims to develop a pipeline/neural network to identify if a person in the scene is talking to or looking toward the camera, and ideally, to generalize towards identifying conversation partners. Your approach will be benchmarked against a well-defined baseline method. Such problem will advance the capabilities of egocentric-based models in social robotics and HCI applications.

 

Your profile :

Know-how

• Currently pursuing or recently completed a master's degree in Computer Science, Data Science, Electrical Engineering, or a related field.
• Familiarity with signal processing: missing data, filtering, etc. Proficient in programming languages such as Python, C++, or Matlab
• Familiarity with computer vision and video processing techniques.
• Expertise in field of Neural networks and specifically transformers architecture.
• Experience with deep learning platforms (PyTorch, or TensorFlow; weights&Biases).
• Knowledge in any of the following is a plus: Encoder/Decoder type Architecture, multi-modal AI, TransFusion, audio and NLP.

Interpersonnal skills

• a good team player ready to tackle technical challenges
• experience with vital-signs monitoring, posture and human motion algorithms is a plus.
• Highly self-motivated with the ability to independently drive projects.
• Capability to autonomously learn new technologies and apply them creatively.

CSEM mission and values

Our mission is the development and transfer of innovative technologies to the Swiss industry. Our objective is to make an impact on our customers and on society at large in the fields of precision manufacturing, digital technologies and sustainable energy. Our strength is the excellence of our people, about 600 passionate specialists dedicated to innovation and technology transfer. We believe that strong values support the successful development of our organization as well as the harmonious and balanced development of all our employees.

We are

• A unique place between research and industry at the cutting edge of new technologies
• An innovative, non-profit, and employee-driven company
• A dynamic, multidisciplinary, and multicultural environment
• A solar team focused on enabling solutions to energy challenges for a sustainable world

Working@CSEM means

• being part of a passionate community
• incredible flexibility, attractive working conditions, and great opportunities of development
• benefit from a management style based on trust & feedback and that favors a work-life balance

We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity.

We look forward to receiving your complete application file via (CV, cover letter, certificates & diplomas) our job page.

Preference will be given to professionals applying directly.