Mediated Reality Mirror

Mediating the Autistic and Non-Autistic Worlds

Open offices are challenging work environments due to their distracting, social and immediate nature, and they are even more burdensome for those with cognitive impairments, particularly with autism. Over the last decade, companies from different fields such as software development (e.g. SAP, Microsoft) have started employing autistics after recognizing valuable work output that autistics create by their information processing skills, attention to detail, preferences in repetitive tasks and trustworthiness. However, their unique way of communicating and interacting with their surroundings, perceiving and processing information requires (1) special training for their coworkers, (2) modifying the work environment to make it more organized and less distracting and overwhelming, and (3) a tutor to moderate the social relations, the tasks and the idle time activities. These conventional solutions increase autistic employees’ work performance to some extent, yet they are not time- and cost-effective, are hard to integrate in different work environments and fall short to support natural social interaction between the coworkers, and the workflow. To overcome these fundamental issues where autistics work, here we present a novel concept, Mediated Reality Mirror (MRM), that enhances reciprocal interaction between autistic and non-autistic coworkers while fostering autistics’ work performance and work inclusion in open offices.

Autism spectrum disorder (ASD) is a wide range of neurodevelopmental conditions characterized by problems in communication and social interaction, sensory-perceptual atypicalities and repetitive and restricted behaviors. The manifestation degree of ASD varies from person to person and it is a lifelong condition which cannot be uprooted from the individual even with early intervention. The global prevalence of autism is estimated to be 62/10000 with an increasing trend, and with an unemployment estimation up to 75%, while the rest struggles sustaining a fulfilling employment due to numerous internal and external obstacles: Their condition requires tailor-cut vocational training and tutoring which are usually neglected by companies. Difficulties understanding directions, social cues, rules and roles, interacting with their coworkers hinder their performance and may result in job termination. Sensitivity to visual, social and auditory stimuli such as lighting, noise, movement, change, social interaction, as well as problems in executive functioning such as motor planning, sustained attention and acclimation to novelties and unexpectedness adversely affect successful task execution. Behavioral atypicalities such as tantrum and aggression may be misinterpreted or not well tolerated by their coworkers which results in high level of stress and anxiety while trying to socially fit in a non-autistic environment. Thus, implementing novel solutions in work environments to improve the abilities, experiences and potentials of autistics as well as their coworkers is a crucial task, which may also impact autistics’ vocational opportunities.

Standing at the intersection of human-computer interaction, interactive information visualization and cognitive psychology, MRM offers an assistive and adaptive work environment for supported and even competitive employment for individuals with ASD. MRM is a spatial and half-reflective natural interface which regulates the visually and socially challenging factors. It diminishes the distractions from the environment while augmenting social and task related visualizations, prompts and cues depending on the real-time sensory data (e.g. availability, emotional state, proximity between colleagues) primarily collected from the autistic employees and secondarily from their coworkers. The physical setup of MRM is highly affordable and consists of a wall (any wall in an open office), depth-sensors, a projector and a processor for real-time image manipulation. The preliminary prototype of MRM already functions in terms of user detection and basic image manipulation. 

Given autism is a lifelong condition which can only be improved on the functional level, MRM pursues to serve as an assistive, but more importantly an adaptive platform which provides a common ground for individual differences, rather than taking them to a normative level. Taking this neurodiversity perspective, this project aims to make the following contributions: (a) exploring design patterns and user experience dimensions for adult autistics, who have been long neglected by HCI due to considerable orientation in early intervention technologies, and (b) using sensory data to inform the design process as well as to create a medium tailor-cut to individual.


Details:

Imagine yourself working at your desk in a crowded room sitting in front of a huge-sized mirror-like wall where you can see your own reflection and the people together with the space behind you. This wall is actually a projection display. It works with one depth-sense camera that captures the live scene of the room including your body and face. By means of a software, the captured image is transmitted to a projector which then projects it to the wall continuously so that you perceive the image as if it is a real reflection. Apart from reflecting the real time image to the wall, this display serves as a mediated reality system. It recognizes you and refines the image according to your social, emotional and perceptual needs in that crowded room.

Functions:

Diminishing the reality: Most of the movements, colors and forms you see around you, distract you. So, the display (that reflects the visual surrounding) erases specific components from the real time image and fills the erased area according to the properties of the surrounding area. The components that can be erased: People who are walking around // The person standing behind you // The person with a yellow t-shirt // An object with an organic form (in other words, no geometrical forms such as cubes, prisms, cylinders etc.) // Everything in the room apart from you

Modifying the reality: In some cases, even if they annoy you, you still need to see some of the distractions. So, the display modifies them in a way that they don’t annoy you anymore. In other words, it takes the annoyance out of these distractions. The possible modifications are that can be applied on the distractions are:

That person’s yellow t-shirt turns into another color (black, red, blue and green) // The giant close-up photo of a happy face on the side wall becomes abstract // Another painting with a checkered pattern on that wall becomes striped // Everything in the room apart from you becomes blurry or abstract

Regulating the proximity: Sometimes you don’t want any social interaction while working and sometimes you need some personal space to interact with people around you. So, the display helps you work by regulating the proximity between you and the people. These regulations are:

Showing your state of mind with abstract animations in a way that other people don’t come close to you or bother you // Animating how much personal space you need or how much they should get close to you

Keywords: Autism; interaction design; attentive user interface, mediated reality; attention, communication

Team: Doğa Çorlu, Asım Evren Yantaç, Morten Fjeld

Contributors: Sıla Kabil, Ilgaz Çiftçi, Beste Özdeşlik, Abdullah Siddique, Anam Tahir, Pelin Karaturhan, Utku Evci

Publishing:

forthcoming: Corlu, D., Tasel, S., Turan, S. G., Gatos, A. & Yantac, A. E. (2017, June). Involving Autistics in User Experience Studies: A Critical Review.

Corlu, D., Ozdeslik, B., Yantac, A. E., & Fjeld, M. (2016, October). Mediated Reality Mirror: Towards a Study with Autistic Users. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (p. 108). ACM.

Yantac, A. E., Corlu, D., Fjeld, M., & Kunz, A. (2015, September). Exploring Diminished Reality (DR) Spaces to Augment the Attention of Individuals with Autism. In Mixed and Augmented Reality Workshops (ISMARW), 2015 IEEE International Symposium on (pp. 68-73). IEEE.

User Resesarch 70%
Ideation 40%
Prototyping 60%

April 2017

Our critical review of UX studies has accepted as a full paper by DIS 2017 conference!

We conducted a systematical review of user experience studies with autistic users conducted between 2010 and 2016. The aim of this study was to reveal how autistic users from any age group are involved in UX studies which focus on developing HCI solutions for this specific user population. We mainly focused on the methodological approaches and practices for conducting studies with autistic users and explore the nature of these studies. Following an iterative critical analysis approach comprised of 4 steps (retreiving papers, choosing appropriate papers, thorough elimination and analysis) we analyzed 98 published studies from a wide range of disciplines.

With this study, we identified the major problems encountered in the literature, b) guidelines for conducting UX studies with autistics, and c) how to report these studies for better development of UX techniques.

Contributors: Şeyma Taşel (method development, qualitative analysis), Semra Gülce Turan (paper review), Athanasios Gatos (quantitative analysis).

15 November 2016

We published a call for collaboration with engineers. If you are an engineer and interested in this project, we are sure we can find a way to work together!

23 – 27 October 2016

We presented our poster on Mediated Reality Mirror informed by the Bodystorming workshops @NordiCHI 2016 in Göteborg, Sweden.

SUMMER RESEARCH PROGRAM

1. Research

 

During the first week of the program, in order to get to know each other, you will be expected to make a brief introductory presentation including your background and the studies you’ve done so far. We will present our studies and our work environment as well.


Literature Review & Archiving

In order to understand what we are working on, you will be losing yourself in attentive user interface (AUI) literature. You will also learn some basic knowledge about sensors and data collection regarding to the project. We will provide you a folder where you can find related resources before you start the program and you will be expected to expand its content. Individual research on the topic beforehand is also welcome!


Meeting

We will arrange a meeting with two of the project contributors in order to discover the existing Refined Reality Mirror prototype. They will explain both the technical details and interaction implications gained through a previous workshop series. You will be expected to ask and discuss about the prototype since you will work on it throughout the summer program.
guests: Sıla Kabil, Beste Özdeşlik


time: Week 1

team: Anam Tahir, Bakhtawar Abbasi

2. Ideation


Workshop 1

You will organize and conduct a collaborative workshop where you will present your ideas about the project  explaining how you think you would contribute to the development of the prototype. These ideas will be expected to include 3 main aspects of the prototype: 1) image processing, 2) application of visualizations and 3) application of interaction models for the prototype. Regarding to your interest, you will focus on one of the aspects.

time: Week 2

team: Anam Tahir, Bakhtawar Abbasi


Workshop 2

You will be expected to attend our bodystorming workshop either as a participant or an observant. The outcomes of this workshop will inform the development process of Refined Reality Mirror. Details of this workshop will be announced on Events page.

time: Week 2

team: Pelin Kömüroğlu

3. Prototype


Image Processing 

You will create the visual input system which will use a) user/face recognition b) emotion recognition c) proximity d) action detection (including working on table, standing up, active area recognition) and e) stimulus (human, non-human) detection.

time: Week 3  7

team: Airas Akhtar, Bakthawar Abbasi


Application of Visualizations

You will code the user interface where the emotion and proximity visualizations are displayed. We will provide you the visualizations created during the previous studies and workshops with animation and graphic designers. In addition to ingtegration of the visuals into the prototype, you will be expected to develop these visuals according to feedbacks. On the other hand these visuals should be categorized in the code properly to make it easier to use in the user studies and applying the interaction models.

time: Week 3  7

team: TBA


Application of Interaction Models

You will code the interaction models (which are explained during the first meeting) for the prototype. Working with image processing and visual adaptation teams, you will be expected to combine their works and apply for at least 4 attentive interaction situations.

time: Week 3  7

team: Anam Tahir

3. User Studies (US)


US 1: Mirror Experience

You will conduct a sensory ethnography study on mirror usage in daily life. This study will be expected to include different places where people use mirrors for different purposes.

US 2: Open Office Experience

You will conduct a sensory ethnography study on work environment experience in open offices.

US 3: Refined Reality Mirror Experience

You will spend a whole day using the Refined Reality Mirror prototype and record your expereinces.

time: Week 3 7

team: Pelin Kömüroğlu


Report

You will write a detailed report of the studies you did throughout the summer. You also present these studies to all of us during the last week of the program.

time: Week 7

team: Anam Tahir, Bakhtawar Abbasi, Pelin Kömüroğlu

14 April 2016

What a spring break! We have just submitted our Bodystorming Workshop outcomes in NordiCHI 2016. Announcements are in June, looking forward to read the comments.

2 April 2016

We conducted two consecutive Bodystorming Workshops last week. The participants played the charades using a mirror and various craft materials in order to enact and visualize a distracting scene to the audience. Afterwards they all met with the Refined Reality Mirror prototype for the first time. Their future projections for such an interface was astonishing. We have already started writing the outcomes of the workshops and prototype trials. Thank you for playing with all the craft materials, communicating through the mirror and sharing your thoughts on the prototype!

An outstanding bodystorming and visalization example from the workshop.
An outstanding bodystorming and visalization example from the workshop.

22 March 2016

Pelin is working on the emotion visualization for Refined Reality Mirror. She’s using tracing paper to work on different emotions on different facial emotions. Some good examples are below.