Considerations when designing future multisensory interfaces
Photo by Heather Oughton on Flicker is licensed under CC BY 4.0
Bluebells are described as being impossible to paint on a canvas. Could they ever be rendered into a mixed reality experience of a near future Multisensory-Enabling Technology? Even if you could paint a digital realistic likeness, would you be able to reproduce their heavenly scent? Or the dappled light that may fall on your face as you walk through the woods, listening to bird song? You may well ask why would you want to when you can experience for real? There are others who may not be so fortunate as you are, unable to venture outside to experience the fresh air. We take so much for granted when we are able bodied, do not live in inner cities and are not stuck in high-rise blocks of flats. Others experience isolation in different ways, such as living alone, lacking confidence to travel or lack mobility or the means to fund travel. In attempting to recreate what seems impossible does that it provide equality of access? Or does it continue to exclude by the technology requirements to access the simulations and their associated cost and needs for internet access? All because you can create a simulated digital environment for learning – will it be better than the real thing?
Look at the poster below from a near future. How would you feel about encountering simulated fire or pain? Is this something for the good if it saves lives in the long game?
What would you fear? How would your reactions be monitored and who owns that data?
In ‘Framework for affordance in Virtual and Augmented reality’, Steffen noted VR can have disadvantages, and the lack of haptic richness found in the physical world that gives a vividness (Slater and Wilbur, 1997). If a certain experience requires high haptic richness and VR technologies struggle to recreate that, the sense of presence will suffer’ (Steuer, 1992). Steffen looks to the future that the gap between the virtual and physical will be a less of a concern and that ‘Perhaps VR’s best uses are those where knowledge or experience gained outside physical reality’s context can later be used to bring benefit to the physical world, whether through some learned skill or through an improved emotional or mental state’. (Steffen, et al., 2019) (p. 690). Future technology is closing in, is it too late to retain the physical world as we know it?
In the Anatomy of Touch (BBC, 2020), research has indicated that the brain does not need much digital input to stimulate the brain to reconstruct memories. The unintended consequences may cause panic even post traumatic stress disorder.
Imagine being placed in a learning environment developed by others and there is no understanding of the participant’s life history. Could it trigger unpleasant past experiences? What about the ethics if the developer’s virtual reality is flawed or out of context for the participant’s physical reality? In a simulation of dangerous environment such as a fire or accident is it possible to account for all the variables that can occur? Who owns the user data generating in this environment? Can commercial bias, persuasion or manipulation be an issue?
This may seem extreme, however in designing complex systems to simulate for example Health and Safety incidents, a wide profile of the needs and experiences of the participants should be investigated. And if necessary escape mechanisms should be incorporated whether via artificial intelligence or by human intervention.
As the author of this OER, I believe there will be a tremendous impact on future education of multisensory interfaces. I believe it will open a new window to the world to those who cannot see, touch, hear, taste, smell what is the ‘real’. It will simulate impossible or dangerous situations and in doing so may save lives or stimulate imaginations. There may be issues with affordances in creating the most ‘realistic’ of real. There will be divides, where inequality of access is inevitable. But there will also be unity where participants can experience in virtual reality what they can not in the physical world.
However, I believe there is a limit to reality, even a pointlessness in some solutions for problems that may not even exist. I believe there should be more design considerations as the brain is a phenomenal computer and the cognitive power will make connections from the depth of memories of the past. That may cause ‘lovely’ sensations, and it may cause horror!
So why overload when you could digitally detox in the big outdoors? Let’s hope future technologists never learn to render a bluebell – somethings are just perfect as they are!
As you have read in this OER, Virtual Reality, Augmented Reality and Mixed Reality are rapidly evolving. Multisensory interfaces in the near future will have haptic capability as well as olfactory and taste possibilities to make virtual experience more realistic. The objective of this course is to ask the participant to think about the future implications of the integration of multisensory interfaces in education. It is hoped you now have an awareness of what is currently available and consider whether there is potential to use what is already present in education or not. It is hoped you can form views on what future technologies you would like to see in education contexts and maybe what should be excluded.
It will be interesting to return to the introduction and ask yourself again if there ‘Can there be too much reality in a simulated digital environment?’ and see if your views have changed!
Thank you for your participation
You have now completed this Open Educational Resource!
Special thanks to Dr Jen Ross, Dr Huw Davies and
Fellow students, Ana Lopes and Adrian Birch for their peer review, advice and support
Digital Futures for Learning
MSc in Digital Education at University of Edinburgh 2020
Ashton, K., 2009. That ‘Internet of Things’ Thing. [Online]
Available at: http://www.rfidjournal.com/articles/view?4986
[Accessed 03 11 2020].
BBC, 2020. Anatomy of Touch: Digital Touch. [Online]
Available at: https://www.bbc.co.uk/sounds/play/p08tn4w8
[Accessed 2020 11 22].
BBC, 2020. The Touch Test: The Results. [Online]
Available at: https://www.bbc.co.uk/programmes/m000n5xx
[Accessed 04 11 2020].
Dreyfus, H. L., 2008. On the internet (second edition). New York: Routledge.. New York: Routledge.
Dufva, T. & Dufva, M., 2019. Grasping the future of the digital society. Futures, pp. 17-28.
Facer, K., 2016. Using the Future in Education: Creating Space for Openess, Hope and Novelty. In: H. Lees & N. Noddings, eds. The Palgrave International Handbook of Alternative Education. s.l.:s.n., pp. 60-77.
Fortunati, L., 2017. For a dynamic and post-digital history of the Internet: a research agenda. Internet Histories – Digital Technology, Culture and Society, 1(1-2), pp. 180-187.
Johnson, S. & Coxon, M., 2016. Sound can enhance the analgesic effect of virtual reality. Royal Society Open Science, pp. 1-6.
Pranav, P., Patel, S., Patel, N. & Shah, M., 2020. Systematic review and meta-analysis of augmented reality in medicine, retail, and games. Visual Computing for Industry, Biomedicine, and Art, 3(21), pp. 1-20.
Steffen, J. et al., 2019. Framework of Affordance for Virtual Reality and Augmented Reality. Journal of Management Information Systems, 36(3), pp. 683-729.
Steuer, J., 1992. Defining virtual reality: Dimensions determining telepresence. Journal of Communication, p. 73–93.
Ultraleap, 2020. Touch is going virtual. [Online]
Available at: https://www.ultraleap.com/haptics/
[Accessed 4 11 2020].
Welty, C. & Setiawan, A., 2019. Digital Immersion – case studies in virtual reality. SHS Web of Conferences, 44(01007), pp. 1-20.
Wilmot, J., 2019. What I Learned from Spending a week in Virtual Reality. [Online]
Available at: https://www.youtube.com/watch?v=eX2QBlckPnw