Keywords:

mixed reality, online learning, metaverse

The world is facing new challenges in the digital era, especially when it is presented with the metaverse and other online platforms. The sector that is affected by this challenge is the education sector. Academics are faced with the challenge of keeping up with digital developments toward society 5.0. The challenge is in the form of online learning. The use of software in online learning still benefits for learning that is theoretical but it is ineffective in practical learning due to unsupported technology. With the emergence of Augmented Reality and Virtual Reality technology, it is possible to make online learning by combining AR and VR technology, namely Mix Reality. Mix Reality provides enormous potentiality for online learning because it provides features that cannot be provided with conventional online learning in general. The combination of haptic AR, audio AR, and gustatory and olfactory AR makes online learning possible. Its application in online practicums, MR technology, has a crucial role because teachers can use MR to produce digital content with no restrictions and interact online with their students. Either through virtual spaces or a collaborative learning environment. This article discusses details about the potential of mixed reality for online practicum learning.

INFO ARTIKEL

ABSTRAK

Kata kunci:

mixed reality, pembelajaran online, metaverse

Dunia menghadapi tantangan baru menuju era digital, terutama ketika dihadapkan pada metaverse dan platform online lainnya. Sektor yang terkena dampak tantangan ini adalah sektor pendidikan. Akademisi dihadapkan pada tantangan ini untuk mengikuti perkembangan digital menuju masyarakat 5.0. Tantangan yang dihadapi adalah pembelajaran online. Penggunaan software dalam pembelajaran online masih bermanfaat untuk pembelajaran yang bersifat teoritis, namun kurang efektif dalam pembelajaran praktikum karena belum didukung teknologi. Kemunculan teknologi Augmented Reality dan Virtual Reality memungkinkan untuk melakukan pembelajaran online dengan menggabungkan teknologi AR dan VR, yaitu Mix Reality. Mix Reality memberikan potensi yang sangat besar untuk pembelajaran online karena menyediakan fitur yang tidak dapat dilakukan dengan pembelajaran online konvensional pada umumnya. Kombinasi Haptic AR, Audio AR, gustatory, dan olfactory AR memungkinkan pembelajaran online. Penerapan dalam praktikum online dan teknologi MR memiliki peran penting, karena guru dapat menggunakan MR untuk menghasilkan konten digital tanpa batasan dan dapat berinteraksi secara online dengan siswa, baik melalui ruang virtual atau lingkungan belajar kolaboratif. Artikel ini membahas detail tentang potensi Mixed Reality untuk pembelajaran praktikum online.

Introduction

Human life today has explored "society 5.0," which means that people are familiar with computers and have used information technology that can be accessed anytime and anywhere. The use of information technology is the beginning of human life's quest for a better future. The world of technology began to refer to the digital world, which is better known as the "metaverse." The metaverse is a 3D virtual network that can be accessed via a virtual reality headset. The development of the metaverse can create new forms of interaction that users cannot have in the real world. Similarly, businesses use the metaverse to hold virtual or virtual meetings. With this technology, people can interact anytime and anywhere in real time without having to leave home.

The sector that can be affected by the presence of the metaverse is the education sector, especially in practical learning methods. Online practicum learning is a challenge for educators today due to the lack of access to metaverse technology. Metaverse itself has begun to be built by users to build their digital platform. However, in practice, online learning still has drawbacks, especially for practical learning. It is necessary to link AR (augmented reality) and VR (virtual reality) technology to make online learning more effective. This combination is known as mixed reality, or MR.

Online learning provides great opportunities for developers to develop MR content to meet human needs in conducting the learning process. However, the obstacles that have occurred are that practicum learning requires physical skills and experience to master the practicum. With limited software that cuts the learning path because it cannot physically touch the object, MR technology development needs to be done to hasten the practicum learning process. This article will examine the potential and discuss the use of MR technology to meet the needs of practicum learning.

Method

The method to be applied in this research is a qualitative research method with a literature review approach. The data collected is useful data from Mixed Reality and its application in other applications. By studying the process and its application in other applications, the author can identify the potential and see the opportunities that exist with the development of technology now or in the future. These opportunities can then be applied to the online learning process using mixed reality. The phenomena that have occurred will be re-examined to add new data that can help in research. To add data, the authors also read journal articles about the potential of MR technology and its applications.

Result and Discussion

Mixed Reality, Augmented Reality, and Virtual Reality

Before discussing MR, the author wants to discuss augmented reality, commonly abbreviated as AR. AR is a technology that displays direct or indirect 3D digital objects generated from computers with various kinds of information combined in the real world (Google Developers, 2022). AR allows you to add fields to the real world and have them run interactively and in real time so that they look like real objects. It is a natural way of exploring 3D objects and bringing these objects into real life. This AR potential supplies information in the form of object visualization, commonly used in advertising, navigation, learning, and other forms. AR is also easy to access using portable digital devices such as smartphones or other forms of AR glasses (Furht & Carmigniani, 2011).

In its application, AR technology can be accessed and used in education, business, agriculture, and other sectors. This is because AR itself can be used as a tool for collaboration. Users do not have to be in one place to interact with each other. Users can meet online and appear as if the person is in the same room. Figure 1 is an example the use of AR for collaborating online and in real-time at the same time.

Figure 1 Online Collaboration in Realtime

On the other hand, virtual reality, or VR, is a simulation of a digital world created by computers to create a realistic world (Ding, Li, & Cheng, 2020). VR is a virtual world where everything is digital. Unlike AR, VR does not involve the real world in its virtual world. Developers can create a virtual world free of physical objects and have no restrictions when creating digital content. This digital simulation creates a real-time 3D simulation where the computer generates all visual senses. Apart from this, there are also the senses of hearing, touch, style, motion, and taste, also called multi-feeling (Speicher, Hall, & Nebeling, 2019). The computer reads and processes data according to what the user does, then enters it in real time. This application makes virtual spaces more realistic and allows for better interaction.

Mixed reality, abbreviated as "MR," is a hybrid of augmented and virtual reality. Experts find it difficult to define a definition of MR, which has conflicting questions. Experts ask whether MR could be a form of AR with more powerful features, or whether it could be VR and AR in one digital device (Pan, Cheok, Yang, Zhu, & Shi, 2006). An example of an existing digital device is Microsoft's HoloLens, which is being developed to meet industry demands. Figure 2 shows an example of using AR to design 3D characters, allowing the user to see her design in real time.

Figure 2 HoloLens, MR Devices by Microsoft

The concept of "mixed reality" brings digital objects closer to becoming part of physical reality. The user is still included in the real world, but it is combined with the virtual world and AR objects. With this combination, VR allows users to interact. Meanwhile, AR makes it possible to bring up digital objects in the real world. When combined, they will create realistic objects in nature, and users can interact with them. MR is the most viable technology platform to replace physical objects and has greater potential in teaching, training, and simulation (Cabrera, et al., 2002). MR does not always use digital graphics to be implemented in the real world, but other forms such as sight, smell, and taste can also be used. The following are the types that will be divided into the classification of AR (Furht & Carmigniani, 2011).

• 1. Visual AR is an AR type that uses visuals. Because it is simple to use, visual augmented reality is commonly found in everyday life.AR visually uses digital devices such as smartphones, AR glasses, and other devices.
• 2. Audio AR does not use digital graphics; it uses auditory sense and is implemented in real life (Mariette, 2012). In a sense, AR Audio introduces artificial sounds made by computers or AI (artificial intelligence) to be implemented into real life (Burdea & Coiffet, 2003).
• 3. Haptic AR, from the word "haptic," which means the feeling of touching. Haptics is the manipulation of objects in the form of touch. Haptic appears when the user has an interaction with the object it touches. An example of haptics that is often used is the vibration of a smartphone. When interacting with the UI on a smartphone, this haptic feedback will begin to be felt by users as feedback from the smartphone itself. However, to deepen the use of haptic feedback, a special tool is needed in the form of VR gloves.
• 4. Gustatory AR and olfactory AR are types of AR that simulate the sense of taste and smell, respectively. This type of AR is still in the research stage, and researchers are still looking for ways to simulate the senses of taste and smell.

Application of MR in Real-World Usage

In industry, MR-based applications have been used. With the industry's desire to use MR in its operations, it has become an important reference in the industry's development. For example, in 2016, Microsoft announced an example of MR devices that were used in a game called the X-ray Project (Figure 3). In this game, the user can display several types of weapons, as shown in Figure 4, where the user displays a digital shield in the game. By using sensors in HoloLens, the computer can detect and recognize the objects it sees.

Figure 3 Project X-Ray, Games in Microsoft HoloLens

In the health sector, Oslo University Hospital has turned 2D images into augmented reality model images. Surgeons can visualize what they meet, and HoloLens helps to navigate within these AR model objects (Kramer, 2017). Apart from HoloLens, software companies such as Spatial.io are also developing digital software in the form of virtual space as a place to collaborate and discuss online and in real-time. This software makes it possible to collaborate directly in a virtual world or each other's place using AR. The virtual space was created to support interactive learning methods that allow users to share images, ideas, concepts, and other forms of media.

Spatial software was created to solve a problem that arose during the COVID-19 pandemic, where users were unable to meet each other in person. In addition, this software is also made to support online meetings using virtual workspaces, which may in the future become a trend in the industry, especially with the presence of the metaverse in society 5.0. Spatial can be accessed via any digital device, including smartphones, the Oculus Rift (VR headset), and the web. With the presence of different platforms, users can access the software wherever and whenever they have an adequate internet connection.

This model of MR technology has a variety of different approaches. Microsoft envisions this AR access using digital devices, namely HoloLens. While Google has a vision of implementing its MR through ARCore, Spatial.io implements it in the form of digital software that can be accessed via an application or interactive website. However, it should be noted that one important thing that the author can conclude is that to be able to apply MR to human life, access to the software is the key to making this technology usable among the wider community.

Potential and Application

During the COVID-19 pandemic, academics were given challenges in facing online learning. Applications in online learning can be made using digital devices that use online learning software such as Moodle, Google Classroom, Microsoft Teams, and other forms of software. The purpose of unifying conventional learning with learning using technology is to help students' learning processes. However, not all online learning processes can be replaced by online learning processes, especially with the practicum learning method. With the presence of technology such as MR, practicum learning will be easier.

The obstacle to learning online practicum today is that students cannot have experience in creating/ experimenting (learning by doing). Existing online learning is limited to two-way communication and virtual interaction spaces (Moodle, Google Classroom, and Microsoft Team). Due to the limited tools available to students, this learning method is insufficient for practicum, and the learning experience, including the trial-and-error process, cannot be obtained as easily as with collaboration. Collaboration in learning is important for students to help each other and complement each other's knowledge when conducting projects or assignments. When discussing real experiences with friends, student involvement is essential.

Digital devices produce digital products, and digital content can be created indefinitely. Content can be created and developed to learn, for example, photography techniques. Computers can make "digital" cameras that can be displayed in the real world through MR devices. There is also necessary information, such as settings in the camera, which can be adjusted in this way. MR can create all of the learning contexts and make it appear as if the actual photographic object is in front of them.

MR has enormous potential to fill the current gaps in the use of online learning. With MR, educators can use it to create useful digital content as if it were real and interactive. Teachers can create learning content, which students can use by utilizing their respective digital devices. Figure 4 depicts the use of augmented reality to teach the anatomy of the human body. This can make it easier for students to be able to see and explore directly using their respective digital devices.

Figure 4 The Human Anatomy Atlas App in AR. The use of learning content from digital devices.

Photometry can use MR technology since it is a course that requires students' skills to develop techniques with the learning by doing method. Photography lectures cannot be obtained from theory alone, but it takes the skills of each student because practicum learning can provide new experiences for students so that they can practice their respective styles.

In online photography learning, providing an interactive website is an alternative that can be used so that students can experiment with different camera settings. A prime example is a website from Canon where students can try various kinds of settings and immediately get the results, as if they could directly see the results of the shot in Figure 8. However, because interactive learning is limited to trying several types of settings, students' skills in determining composition or point of view in photography cannot be applied. To cover this shortcoming, MR provides opportunities for students by displaying a virtual camera view using MR, and then students can try various kinds of settings using a remote controller. This application method is the same as when playing MR games in figures with HoloLens. However, the object displayed is the camera.

The object of photography can also be displayed at will in the learning process because MR can build any object on our device. If we are photographing landscapes, then digital scenes can also be made to support the process. It is the same with other objects, such as shooting models, still life, and others. In its application, as shown in Figure 5, NASA uses MR to be able to simulate the atmosphere of Mars using Microsoft Holo Lens to study the landscape of Mars and collaborate live with others.

Figure 5 NASA, Interaction inside Virtual Reality

Conclusion

In its application, MR can be applied to distance learning through one platform or application, such as an online spatial platform. This platform, which has the potential to facilitate the learning process, is very suitable if it is done through applications or software on digital devices. In the following, the author will express the advantages of digital software for the application of this online learning, namely:

• 1. Digital devices can be used and accessed by the public. Following the development of society toward "society 5.0," people have started to know computers and will know computers throughout their lives. This means that there are digital devices that are widespread and can be used by people of all ages, especially now that technology companies have started to develop their own VR and AR glasses for widespread use.
• 2. Digital devices have a variety of sensors that meet the user's needs. Sensors in digital devices have various functions, such as detecting the presence of light detection and ranging, better known as LiDAR. This sensor can be used for spatial mapping because it detects objects and objects remotely with laser light and radio waves. Not only LiDAR, but technology companies such as Google are also developing AR Core technology (Google Developers, 2022), which uses machine learning to measure space and its environment. The result is a unification of virtual objects and real spaces that can be enjoyed on our digital devices.
• 3. Digital devices enable collaboration, meaning that online collaboration can be done with different digital devices. As in the metaverse, users can meet and interact in real-time. It is the same with current online applications such as Zoom, Google Meet, or Microsoft Team, but the difference is that in these applications, users can only enjoy them in the form of a screen display, not interactively.

Since the introduction of smartphones into human life, public access to digital devices has been implemented. This means that access can be made anywhere. Through the application, developers may be able to modify the process of this learning method. Coupled with the presence of digital device technology that supports MR applications, new experiences can be presented to users, both in learning and direct online interaction. MR technology is presented to meet human limitations by holding activities separately and presenting them on one platform. New learning methods are likely to be applied to society using MR, particularly when interacting in real-time as we do in the real world.

Using mixed reality in distant learning facilitates practical learning since practicum requires skills and expertise with the "learning by doing" method to gain knowledge. This approach is extremely rare if it is done online at first; it differs from theoretical learning, which does not involve "learning by doing." However, using technology-enhanced learning approaches such as augmented reality, virtual reality, and mixed reality, students may be able to seek new experiences in learning that require hand skills and other experiences connected to physical mobility. Mixed reality, augmented reality, and virtual reality are the most appropriate approaches for learning practicum.

References

Furht, B., & Carmigniani, J. (2011, july). Augmented Reality: An Overview. Handbook of Augmented Reality. (B. Furht, Ed.) 3-46.

Geroimenko, V. (2012). Augmented Reality Technology and Art: The Analysis and Visualization of Evolving Conceptual Models. 16th International Conference on Information Visualisation. IEEE.

Sa, M. D., Churcil, E. F., & Ibister, K. (2011). Mobile augmented reality: design issues and opportunities. Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (pp. 749 - 752). Stockholm, Sweden: MobileHCI.

• Mariette, N. (2012). Human Factors Research in Audio Augmented Reality. New York, United States: Springer, New York, NY.
• Burdea, G. C., & Coiffet, P. (2003). Virtual Reality Technology (2 ed.). Canada: Wiley-Interscience Publication.
• Ding, Y., Li, Y., & Cheng, L. (2020, May 11). Application of Internet of Things and Virtual Reality Technology in College Physical Education. Special Section on Deep Learning Algorithms for Internet of Medical Things.
• Speicher, M., Hall, B. D., & Nebeling, M. (2019). What is Mixed Reality? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Paper 537, pp. 1 - 15. CHI '19.
• Pan, Z., Cheok, A. D., Yang, H., Zhu, J., & Shi, J. (2006, February). Virtual reality and mixed reality for virtual learning environments. Computers & Graphics, 30(1), 20-28.
• Cabrera, A. F., Crissman, J. L., Bernal, E. M., Nora, A., Terenzini, P. T., & Pascarella, E. T. (2002). Collaborative learning: Its impact on college students' development and diversity. Journal of College Student Development, 43(4), 20-34.
• Kramer, S. (2017, April 19). How Microsoft's HoloLens Is Changing Medicine—And Surgery. (Futurum Research) Retrieved April 25, 2021, from Futurum. Technology Insight for Business Leaders: https://futurumresearch.com/how-microsoft-hololens-is-changing-medicine-and-surgery/
• Google Developers. (2022, 02 01). ARCore, Fundamental Concept. (Google) Retrieved July 18, 2022, from https://developers.google.com/ar/develop/fundamentals