SAAC
Issues in surgery can be related to collaboration problems such as miscommunication and misunderstanding. Analysis of these collaborative situations can be done by experts, but it is a long and complex process. Moreover, this approach does not allow for real-time analysis and feedback on the collaboration. Some recent approaches aim to assess and model the quality of collaboration using measures that can be computed in real-time (e.g., non-verbal, gaze, physiological data). Considering some of the challenges related to the evaluation and modeling of collaboration, the SAAC project aims to compute metrics in real-time to report on the quality of collaboration within a group. The ultimate goal of SAAC is for the system to be seen as a collaborating system by the team, able to interact with the team when detecting miscommunication or miscoordination to prevent some adverse events.
Since the beginning of seminal research on multimodal human-computer interaction, much progress has been made in capturing users’ activities to assess various individual and team processes. Indeed, the number of sensors and the general computing power available to researchers have increased substantially. The capture and processing of complex social behavior data such as gaze, gestures, speech and para-verbal data are now possible in real-time. This has contributed to the increase in research on modeling and detecting collaboration using multimodal data over the past decade.
Objective
For efficient collaborative activities, collaborators must have an updated representation of the collaborative situation. To do this, they construct together a shared mental model of the situation. Known as common ground, this representation refers to the sum of shared information about the understandings, knowledge, beliefs and assumptions between two or more people during an activity. The goal of SAAC is to model and share a digital representation of the common ground, so the system can build its ability to participate in the collaborative activity.
Data visualization with the PsiStudio tool monitoring the record of an experimentation session with two users solving a puzzle tasks together in a virtual environment.
full video
Method
Our approach is to apply and test the effectiveness of collaboration indicators found in the literature (cognitive ergonomics, social psychology, management, etc). We conduct this in small groups collaborative situation in a controlled virtual environment to measure collaboration problems in different ways. Using a virtual reality collaborative task has the advantage of collecting precise behavioral data but more importantly, enable fine control of the collaboration situation and the deployment of a variety of feedback in response to users' actions and collaborative states. The application of measures and augmentations of collaboration in virtual reality allows us to test their effectiveness and their effects on the situation before applying them in a real situation.
The SAAC Framwork: The Collaborative Virtual Environment module (top left) contains screenshots of different collaborative situations, with and without collaborative augmentation. The Perception module (a) contains the acquisition and filtered part of the gaze signal. In the Collaboration assessment module (b), the inference model part contains algorithms that allow calculation of joint visual attention events, mutual gaze, and people gaze events reflecting respectively the mutual understanding and part of the team situation awareness. Then, in the Collaboration augmentation module (c), the CVE was augmented accordingly to the cues calculated. (1) show the addition of information that a participant looks at us using a filter colored of the other participant color; (2) show the addition of a cyan particles effect, indicating that the two participants are looking at the same object
Expected results
In the SAAC, we hypothesize that a system observing the team’s work could detect those collaboration issues and eventually provide feedback to assist teams in working together more efficiently. The expected results are validated measurements of collaboration a system can measure, and a software framework.
SAAC is based on \psi, Microsoft’s Platform for Situated Intelligence, and open source
Role of IMT Atlantique
SAAC is an original research project from the PACCE team at IMT Atlantique / LS2N
Contributors to this project from IMT Atlantique were Cédric Dumas, Mathieu Chollet, Alexandre Kabil, Aurélien Lechappé, Arnaud Allemang-Trivalle, Aurélien Milliat.
We collaborate with laboratories such as LIST (Luxembourg), IBISC (Evry), LISN (Saclay), Labsticc (Brest), University of Glasgow and Microsoft Research (Redmond).