INFO ARTIKEL

Kata kunci:

Bahasa asing, evaluasi program, training, realitas virtual, UMKM

ABSTRAK

Artikel ini mengevaluasi integrasi teknologi virtual reality (VR) dalam pelatihan bahasa asing untuk UMKM di Indonesia dengan menggunakan Model Kirkpatrick. Penelitian ini melibatkan pelaku UKM dari berbagai industri yang berpartisipasi dalam program pelatihan bahasa VR yang disesuaikan. Survei dan wawancara dilakukan untuk menilai reaksi, persepsi, dan relevansi peserta terhadap tujuan pembelajaran bahasa. Evaluasi tersebut mengukur hasil pembelajaran, seperti peningkatan kemampuan bahasa dan peningkatan kepercayaan diri dalam menggunakan bahasa asing dalam lingkungan bisnis. Studi ini juga menyelidiki perubahan perilaku yang dihasilkan dari pelatihan VR, termasuk aplikasi keterampilan bahasa dalam kegiatan bisnis sehari-hari dan strategi komunikasi dengan mitra atau pelanggan internasional. Dampak keseluruhan dari program ini terhadap daya saing UMKM, perluasan pasar, dan kolaborasi

internasional juga dievaluasi. Hasil penelitian menunjukkan program pelatihan VR efektif, dengan reaksi positif, hasil pembelajaran, dan perubahan perilaku dalam kegiatan bisnis. Studi ini menekankan pentingnya mengukur reaksi dan menyarankan perbaikan dalam materi pelatihan dan jadwal pelaksanaan berdasarkan umpan balik peserta. Program pelatihan bahasa berbasis VR menjanjikan untuk meningkatkan kemampuan bahasa, daya saing, dan upaya internasionalisasi UMKM.

https://doi.org/10.5614/sostek.itbj.2023.22.3.2

Introduction

In recent years, the small and medium-sized enterprises (SMEs) sector of the global economy have experienced significant growth (Chelliah, 2010; Gherghina, 2020; Palanimally, 2016). The dynamic actors of SMEs play a crucial role in promoting economic growth, job creation, and innovation in numerous nations, including Indonesia. As the world becomes increasingly interconnected, the ability to communicate effectively in foreign languages has become increasingly important for small and mediumsized enterprises (SMEs) to compete in international markets (Kosti-Bobanovi, 2017; Incelli, 2008). The ability to communicate accurately and fluently in multiple languages allows SMEs to overcome language barriers, establish international partnerships, attract foreign customers, and expand their global reach (Lim, 2010; Tenzer, 2017; OECD, 2019). This highlights the significance of language proficiency in enabling SMEs to navigate the complexities of the global business landscape and exploit international opportunities (Tenzer, 2017; Grzeszczyk, 2015).

To flourish in today's competitive business environment, SME actors require ongoing skill development, including training in foreign languages. Traditional classroom-based training has been prevalent, but the emergence of online training platforms offers numerous benefits (Gilbert, 2015; Al Rawashdeh, 2021; Coman, 2021). SME actors increasingly favor online training due to its flexibility, cost-effectiveness, accessibility, self-paced learning, diverse resources, scalability, and opportunities for continuous learning (Klimex, 2009; Roy, 2008; Ulas, 2019). Online training enables SME actors to access materials at their discretion, reduce costs, overcome geographical barriers, personalize learning experiences, and accommodate business growth while remaining current (Dawe 2007; Roy 2008).

Conventional classroom-based methods frequently lack real-world relevance and fail to provide an immersive and engaging learning experience (Liu, 2020). Consequently, establishing the language skills which is necessary to effectively communicate with customers, suppliers, and partners across the globe is difficult for SMEs (Ulas, 2019).

The application of virtual reality (VR) technology is one possible solution to this problem, according to Bonner (2018). By simulating real-world environments and allowing students to practice language skills in authentic settings, virtual reality (VR) technology offers a novel and immersive learning experience (Pack, 2021). VR has the potential to transform language learning by making it more effective and pleasurable due to its interactive and engaging nature (Bonner, 2018; Lin, 2015). The adoption of VR in language training programs for small and medium-sized enterprises (SMEs) in Indonesia, however, it remains relatively unexplored, and its effectiveness and impact must be evaluated.

This article provides a comprehensive program evaluation of the integration of VR technology into foreign language training for Indonesian SME actors. The evaluation is based on the widely recognized Kirkpatrick Model for assessing training programs. The Kirkpatrick Model has four levels: reaction, learning, behavior, and outcomes (Kirkpatrick, 2016). This study seeks to provide a comprehensive understanding of the outcomes and efficacy of the VR-based language training program by employing this model.

Participating in the evaluation were SME actors from a variety of industries in Indonesia who had participated in a VR language training program designed to meet their specific requirements. Through surveys and interviews, the participants' responses to the program, including their perceptions of the

VR experience and its relevance to their language learning objectives, will be evaluated. In addition, the evaluation will assess the learning outcomes of the participants, such as improved language proficiency and increased confidence in using foreign languages in business settings. In addition, the evaluation will investigate the behavioral changes caused by the virtual reality language training program. This includes determination if participants use their newly acquired language skills in their daily business activities and if the training affected their communication strategies with international partners or customers. The study will be concluded by evaluating the overall impact of the program on the competitiveness, market expansion, and international collaboration of SMEs. Specifically, the research topics are as follows:

The evaluation involved SME actors from various industries in Indonesia who had participated in a structured VR language training program tailored to their specific needs. The participants' reactions to the program, including their perceptions of the VR experience and its relevance to their language learning goals, were assessed through surveys and interviews. Additionally, the evaluation has measured the participants' learning outcomes, such as language proficiency improvements and increased confidence in using foreign languages in business contexts. Furthermore, the evaluation explored the behavioral changes resulting from the VR language training program. This includes examining whether participants applied their newly acquired language skills in their daily business activities or whether the training influenced their communication strategies with international partners or customers. Finally, the current study assessed the overall impact of the program on SMEs' competitiveness, market expansion, and international collaboration. More specifically, the research questions are:

• 1) How is the participant's satisfaction with the English language training program?
• 2) How are the learning processes conducted?
• 3) What are the changes in behavior after the participants return to their businesses?
• 4) What are the final results of the participants' activities after attending the training?

Literature Review

Soft skills in Communication in Foreign Languages and Export Markets

Trade requires communication, and language is an essential component of communication. Nonetheless, the vast enterprise-level literature on export and internationalization (Miesenbock, 1988; Dhanaraj & Beamish, 2003; Andersson, 2004; Ruzzier et al., 2006) has generated some systematic and quantitative conclusions regarding the contribution of language skills. Despite this, there is substantial evidence to suggest that these skills are more crucial than many businesses realize. Only a small percentage of companies have conducted exports; of these, most serve only a few foreign markets, selling mainly to domestic customers (Bernard and Jensen, 1999; Eaton et al., 2004; Bernard et al., 2007; Mayer and Ottaviano, 2007). It has long been noted that a dearth of information, risks, and expensive procedures are the primary impediments to exporting and the cause of poor export performance (Katsikeas & Morgan, 1994; Verhoeven, 1988; Leonidou, 1995; Kedia & Chhokar, 1986; Safari & Saleh, 2020; Johanson & Vahlne, 2017; Rua et al., 2018; Narayanan, 2015).

Mastering a foreign language, particularly a global language, is a necessity that cannot be avoided. As one of the international languages and a means of verbal communication, English has permeated nearly every aspect of human existence on Earth. One of the articles focusing on language in the internationalization of companies is that of Crick (1999). It examines the use of language in exporting small and medium-sized British businesses.

The results of a large survey show the conclusion that most companies are aware of the importance of language, but this is not reflected in their daily work or in the recruitment and training of personnel (Crick, 1999). Furthermore, the article explained that the use of foreign languages has become one of the most important elements of corporate communication when working internationally. The context of the live research is to establish skills important for communication with others whose English is not their first language. Related to the role of foreign language skills in international marketing, Swift (1991) posits that the importance of language is greater than much of the latest language-oriented literature we want to trust. Look at how and why language can be an obstacle to communication, and then detail the use of foreign languages in marketing operations.

The majority of businesses are aware of the importance of language, but this is not reflected in their day-to-day operations or in the recruitment and training of employees (Crick, 1999). In addition, the article explains that the use of foreign languages has become one of the most crucial aspects of international corporate communication. The context of the live research is the development of essential communication skills with non-native English speakers. In relation to the function of foreign language skills in international marketing, Swift (1991) asserts that the significance of language is greater than the majority of the most recent language-oriented literature that we are willing to believe. A more substantial way is to examine how and why language can impede communication and then describe how foreign languages are utilized in marketing operations.

The conclusion is that language is the key to attaining market "proximity" for this crucial reason. In addition, according to Elisabeth Lord Stuart, Director of Operations for the US-Algerian Business Council, the dearth of language skills among U.S. businesspeople is a significant barrier to expanding U.S. participation in international markets. Without appropriate language skills within a company, formal business proposals and informal business cues cannot be communicated abroad.

English Training Materials

The English training material is tailored to the English proficiency issues SME actors encountered, such as very limited English communication skills related to product marketing, introducing, and promoting their products to foreign consumers. In business contexts, negotiation and presentation skills are of utmost importance for success. Effective communication and persuasion are fundamental in business interactions, and these skills are crucial for establishing relationships, influencing decisions, and achieving desired results. (Peleckis, 2016)

Strong negotiators are adept at navigating complex situations, resolving conflicts, and identifying mutually beneficial solutions (Zohar, 2015). Skilled negotiators can secure favorable agreements and maintain long-term alliances by actively listening, understanding their counterparts' perspectives, and employing persuasive techniques (Kennedy, 2020). Similarly, presentation skills are essential for communicating ideas, engaging stakeholders, and making an impression that lasts. Effective presentations captivate the audience's interest, instill confidence, and convey messages clearly (Yee, 2014). Skilled presenters use storytelling, visual aids, and audience analysis to create persuasive narratives and influence decision-makers (Dahlstrom, 2014).

Both negotiation and presentation skills contribute to the development of rapport and trust with clients, stakeholders, and coworkers. Trust is the foundation of productive business relationships, and it is cultivated through effective communication, attentive listening, and keeping commitments (Brunner, 2008). Professionals who flourish at negotiation and presentation can foster trust by comprehending the perspectives of others, addressing their concerns, and presenting information credibly and persuasively (Osmievi, 2020; Hanifa, 2018).

In addition, negotiation and presentation skills offer a competitive advantage in the business landscape. In today's globalized and competitive markets, the ability to negotiate favorable deals, win contracts, and deliver impactful presentations distinguishes professionals (Osmievi, 2020). These abilities demonstrate professionalism, self-assurance, and the ability to influence outcomes, increasing the likelihood that individuals will obtain new opportunities and advance in their professions.

Virtual Reality Technology for Foreign Language Training

Various studies have been conducted to find effective methods and models for improving English as a foreign language. The emergence of information and communication technology (ICT) has contributed to significant fields of innovation in language education. Today, language teaching is becoming more exciting and complex, and the role of a teacher or instructor is changing rapidly (Haleem, 2022). One of the ICT implementations that have been widely used in language learning is the use of virtual reality (VR) technology, which is a technology that gives graphic interaction directly (in real-time) with threedimensional models combined with screen technology that can give an immersive impression (like being in the real world) to the user (Schnack, 2018).

Virtual reality makes it possible to simulate the real world in a three-dimensional visualization system and allows user interaction with the model due to the ability to track movement and the ability to act with the product in real-time (Pérez Fernández and V. Alonso, 2015). In the technological development of the present era, two technological media emerged that are now the latest innovations. Virtual reality (VR) and augmented reality (AR). Neuburger(2018) stated that augmented reality and virtual reality are examples of advanced technologies that have an impact on direct perception because these media have the power to unite perceptions of a variety. There has been a very significant increase in interest in the use of virtual reality technologies to enhance language skills, increase cultural awareness, and develop communicative competencies (Parmaxi, 2020; Yudintseva, 2023). Such platforms often focus on teaching different cultural points to students (Cheng, Yang, & Andersen, 2017), promoting grammatical and vocabulary mastery (Johnson & Valente, 2009), and supporting real-time communication to meet a quest. (Thorne, 2008). However, the application of VR technology in the defense of these languages is widely done for formal learning in schools with the aim of further increasing the interest and motivation of students in language learning and also as a form of innovation in learning technologies.

Method

This research used program evaluation methods. The evaluation aims to see how the program is designed, implemented, and beneficial to the parties involved in it. On the implementation of the program, evaluation is meant to seek as much information as possible to obtain an overview of the planning and program implementation. The results of the assessment will be used by the interested parties to decide. In this study, the content or learning material developed is English literacy for SMEs, in particular presentation techniques and negotiation skills.

Kirkpatrick's evaluation model

The evaluation method used in this study is the Kirkpatrick model evaluation, proposed in 1954 by Dr. Donald Kirkpatrick. The Kirkpatrick evaluation method is an evaluation of the effectiveness of the training program, which includes four evaluation levels (Kirkpatrick, 2016).

Level 1: Reaction

Evaluation of participant reactions means measuring customer satisfaction using a questionnaire to measure the reactions immediately after the training program (Albizu et al., 2019; Antonio et al., 2019; Yong et al., 2019). Training or programs can be said to be effective if participants find an activity fun and satisfying, and therefore they will be interested and have the motivation to learn and practice. Participant satisfaction will be studied from several angles: the material provided, material delivery strategies, learning media used, activity schedules, and others.

Level 2: Learning

Learning evaluation is an evaluation of the effectiveness of activities through attitudes, knowledge, and skills. Participants can be said to have learned if they have experienced a change in attitude, an improvement in knowledge, or an improvement in skills. Therefore, to measure the effectiveness of an activity, these three aspects need to be measured. This level of evaluation can also be referred to as an assessment of learning outcomes (outputs) that compare before and after training (Antonio et al., 2019; Hopkins et al., 2018). The measurement of learning outcomes can be done using the following

questions: 1) What knowledge has been learned? 2) What attitudes have changed? 3) What skills have been developed or improved? and others.

Level 3: Behavior

Evaluation at this level is focused on changing behavior after participants return to their businesses. Is the change in attitude being implemented? Can participants transfer the knowledge and skills they have gained? At this level of evaluation, it can be referred to as an evaluation of the outcomes of the activity.

Level 4: Results

This level of evaluation is focused on the final results of participants' participation in an activity. The categories of results of this activity are an increase in sales, a decrease in costs, an increase in profits, and others. The evaluation of this level can be categorized as an impact of the activities that participants have participated in.

English for Business training program for Indonesian SMEs

The English for Business training program was conducted in 2021, enrolling SMEs in Indonesia. It was held over 30 consecutive days. Before starting the training, the participants must complete a pre-test that has been adjusted to the topic to be discussed. The participants were also given an individual project to complete during the training.

The sampling method used in this study was purposive sampling. Since the material is not basic English and focuses on internationalization, the participants selected for this study were those who have basic English capabilities and who have a goal to become exporters. To choose SME actors that will be given training, previous observations were made to see which SME actors should get training to develop their English capability. From the results of observations and initial interviews, 20 SMEs were identified. The participants in this training are SME actors in Indonesia from various types of businesses.

A General Overview of the Application Used in English Language Training for Indonesian SME

The virtual reality-based application designed is called ConfiYou (Fig. 1), which consists of two training sections, namely presentation and negotiation using English, as seen in Figure 2. The app will be connected to the Google Cardboard tool.

Figure 1 ConfiYou Aps (The Virtual Reality-based application)

VR is integrated with Unity: When training participants use the VR app to choose one of the menus (presentation or negotiation), the scene will appear as if they were in the meeting room. In front of the user, on the screen, will appear an avatar, which is a potential investor who wants to negotiate related to the product that the user (entrepreneur SME) has. With the emergence of several questions asked by potential investors, the user must answer those questions, and he will be directed to proceed to the next question until this stage is completed. There are features of video learning in the form of supplies or learning in theory or practice, either in presentations or negotiation techniques delivered by experts.

Figure 2 Architecture of Applications

The application uses a 360-degree camera as a medium to display questions that feel real, supported by the presence of video learning materials such as presentation techniques or negotiation.

With this architecture, the features help the SME actors as the training participants to train their confidence, comfort, and readiness in presenting and negotiating using English.

Based on the user needs that have been analyzed, the features in the application can be presented in a use case diagram, as shown in Figure 3. There are two factors, namely the user and the system. The system will display presentation and negotiation scenes and ask questions during presentations or negotiations through the audience or potential investors. The user will answer questions in stages until they are finished and can end the presentation or negotiation simulation.

Figure 3 Use Case Diagram

Users can choose the presentation or negotiation mode, then make a presentation or negotiation. If the user chooses a presentation, the system will display a scene in the form of an auditorium, and the user's position will be on the stage. Users make presentations according to the time allotted. Then the user answers the questions posed by the audience. If the user chooses negotiation, the system will display a scene in the form of a meeting room in the office. Users answer every question—questions given by potential investors. Users can select video learning features for learning or training materials so that they can make it easier and can open the material first before carrying out the simulation.

Findings

Following are the findings of the researchaimed to evaluate the impact of mentoring activities in the English for Business training program.

1. Reaction

The level of satisfaction of participants with mentoring activities (Level 1) is shown in Table I. Based on Table II, it can be seen that the participants' satisfaction level is in the range of 85%–89%. It implied that the participants have positive reactions and feel satisfied with the mentoring activities because they are aware of the increase in knowledge and abilities through the training. The sub-elements that still need to be improved are the delivery of training material sequentially, the presentation of material by the speaker clearly and sequentially, explanations by speakers using clear voice and intonation, conformity of material to a predetermined schedule, assistance tools, and support rooms. The six sub-elements have a relatively small weight (<85%) compared to the other sub-elements.

Table I Level of Participant Satisfaction with Virtual Reality Technology-Based English Training (Level 1)

After the participants filled out the questionnaire, a focus group discussion (FGD) was conducted to dig deeper into the level of satisfaction of the participants during and after attending the training. This FGD was conducted on the last day of the post-test.

In general, the participants have a positive reaction to the training. Most of the participants described the training as "satisfying" and "fun". Many respondents also give suggestions on how to improve the training; common responses involve the training schedule and the learning material.

• "[…] "Menyenangkan tapi jadwalnya sering bentrok jadi saya sih sering gak ikut pelatihan ini. Untungnya karena pake teknologi VR, jadi kami bisa mengikuti pelatihan ini kapan pun."
• "[…] "The training was soo fun, but the schedule often clashed so I often didn't take part in this class. Fortunately, because we used VR technology, we can take part in this training at any time." (Participant 8)
• "[…] "Beberapa kali saya tidak hadir karena di akhir tahun banyak kegiatan untuk ngejar target. Tapi karena kita belajar mandiri dengan menggunakan teknologi VR, kami jadi bisa mengikuti, dan trainingnya menyenangkan."
• "[…] "I skipped the class several times because the training was held at the end of the year; there were too many activities to achieve targets. However, since we learn independently using VR technology, we can follow along, and the training was fun (Participant 17)

The two respondents above are examples of improvement suggestions regarding the training schedule. It can be observed that many participants found the schedule to clash with their schedule and would benefit from a more flexible training schedule. Besides that, many participants also commented on the learning material. While some participants find the material challenging, many find it too hard for their English level.

• "[…] Karena jarang gitu ada program training seperti ini. Materinya kalau bagi saya pribadi agak susah ya. Sudah lama sekali tidak belajar bahasa Inggris dengan tata bahasa yang baik. Jadi ya agak sulit terasanya ya. Materinya sih asik sebetulnya. Tapi ya begitu deh."
• "[…] "We rarely have a training program like this. The material for me is rather difficult then. It's been a long time since I learned English with good grammar. So yes, it's a bit difficult. The material is cool…. but that's how it is." (Participant 7)
• "[…] Materinya lumayan ya…susah kalau bagi saya yang udah lama sekali tidak belajar bahasa Inggris. Banyak yang lupa."
• "[…] The material is pretty good… it's hard for me, who haven't studied English for a long time. Many have forgotten." (Participant 15)

Common themes suggest that many participants had neglected their English skills and were a bit overwhelmed by the training. Although they acknowledged that the material was very suitable for their needs and directly applicable to their business.

2. Learning

There are two modules given to the training participants in English for business. These two modules are provided according to participants' needs as exporters to add soft skills and lay the foundation for carrying out daily business activities. The five modules are:

• 1. Presenting Products in English
• 2. Business Negotiation in English

To measure how well the participant is able to learn the material, all participants are required to answer a pretest before training and a post-test after training. The pretest and posttest scores are compared to each other. Table X shows the score comparisons of the participants.

Table 3. Knowledge Before and After Training

According to the results presented in Table X, the average score before training was relatively low, between 2.20 and 2.96, and after training it improved; the after-training score was relatively high, between 4.00 and 4.17. This result shows that the participant gained knowledge from the training. In the discussion, participants give positive feedback on the learning experience; most of them report that they feel improvement in their speaking, reading, and writing English.

• "[…]Yang meningkat tentunya skill bicara ya. Tapi skill menulis juga sebetulnya saya merasa ada peningkatan. Memang grammar masih sering salah tapi saya berusaha terus untuk memperbaiki grammar dan biar kalimatnya lebih canggih gitu."
• " […] What has improved, of course, is my speaking skills. But actually, I also feel that there is an improvement in my writing skills. Indeed, I often make mistakes in grammar, but I keep trying to improve my grammar and make the sentences more sophisticated." (Participant 11)
• "[…] banyak kemajuan ya. Terutama saya sih di speaking ya. Jadi bisa ngomong gitu ya meskipun masih dikit-dikit."
• "[…] a lot of progress huh. I'm especially good at speaking. So, you can say I speak English a little bit" (Participant 4)

On top of that, several participants also remarked on how the training gives them valuable experience and motivates them to study English further. Many also admit that the training has contributed to their confidence in using English.

• "[…] Sebelumnya saya sangat takut kalau harus berinteraksi dalam Bahasa Inggris. Saya merasa malu gitu…takut. Sekarang sih saya sudah mulai berani. Mudah-mudahan nanti saya tambah berani untuk berkomunikasi dalam Bahasa Inggris."
• "[…] Before joining the training, I was very afraid if I had to interact in English. I feel so ashamed… afraid. Now I'm starting to be brave. Hopefully, later I will have the courage to communicate in English." (Participants 15)
• "[…] Kemampuan berbicara saya yang terutama meningkat. Saya jadi percaya diri gitu diajak ngobrol pake Bahasa Inggris."
• "[…] My speaking ability especially has significantly improved. I feel more confident when I talk to my potential partners in English." (Participant 1)

3. Behavior

On the third level, the person who is closely related to the participants is interviewed. These people could be their friends, family, and coworkers. The objective of the interview is to find out the changes that happened in the participant's behavior after the training. It will give an insight into whether the training applies to their activities. Interview results show that many participants show positive behavioral changes in their daily lives. Participants immediately applied their newfound knowledge to their businesses; some even taught the knowledge gained to their subordinates.

• "[…] Banyak perubahan setelah pelatihan. Biasanya ibu langsung praktik bicara-bicara pake bahasa Inggris gitu. Ke kita-kita ngajak diskusi pake bahasa Inggris, terus upload-upload produk pake bahasa Inggris. Banyak perubahan lah."
• "[…] Many changes have occurred to my employer after attending this training. He always immediately practices speaking in English after the course, and he often invites us to discuss in English, moreover, he often uploads product videos in English. There are many improvements." (Informant 9)
• "[…] Ada perubahan pastinya. Sering praktik menulis iya, berbicara iya, baca-baca apa dalam bahasa Inggris. Bahkan kadang ngajarin kami di sini, percakapan sederhana aja."
• "[…] There's an improvement. She often practices both her writing and speaking as well as her reading skill. Sometimes she even teaches us here, just simple conversations." (Informant 4)

Many also reported changes in participants' confidence in their English skills; several participants were observed by their relatives to be more confident and proactive in learning.

• "[…] Iya ada. Jadi seperti percaya dirinya ningkat. Dari dulu juga sering sih pakai bahasa Inggris kalau promosi-promosi gitu di medsos, pake bahasa Inggris memang udah biasa. Tapi sepulang pelatihan suka langsung tancap gas."
• "[…] Yes, there has been a change in my boss. His confidence seemed so much increased. In the past, my boss has often used English to promote our products on social media. He is used to using English on his social media. But after training, he likes to step on the gas immediately." (Informant 2)
• "[…] Jadi lebih sering ikut-ikut event, exhibition internasional gitu. Dan jadi lebih percaya diri."
• "[…] Now he is more likely to take part in international events and exhibitions. And he's got more confidence in himself." (Informant 13)

While most of the participants' behavior is positive, a lesser percentage of the participants were reported to not have made significant behavioral changes. Their coworkers usually suspect that the reason for these insignificant changes is related to their frequency.

• "[…] Tidak banyak perubahannya. Kurang kelihatan. Kayak masih sama aja. karena ikut pelatihannya cuman sebentar. Hanya ikut 1-2 kali.
• "[…] Not much has changed. Less visible. It seems that his English ability is still the same as before joining the training. This seems to be because he only briefly participated in her training. He only participated 1-2 times." (Informant 16)

4. Result

The last level in the Kirkpatrick evaluation method is the result. At this level, the learning outcomes are measured at the organizational level. The assessment is conducted through interviews with the participants. In the interview, many remarked that their external communication has improved thanks to the training, while several participants said they gained new international customers after applying for the training lessons.

• "[…] Sekarang lumayan naik jumlah konsumer internasionalnya. Setelah sering ikut event internasional, kan sering itu ikut pameran produk sekala internasional. Jadi lebih aktif ikut buat pameran produk. Dulu konsumernya hanya 1 atau 2. Sekarang lebih banyak…ada lah 4 atau 5 gitu." (Informant 12)
• "[…] Now the number of international consumers has increased quite a bit. After often participating in international events, he often participates in international-scale product exhibitions. So be more active in participating in product exhibitions. In the past, there were only 1 or 2 consumers. Now there are more…there are 4 or 5 like that." (Informant 12)
• "[…] Dulu hanya 1 sekarang konsumer internasionalnya lumayan bertambah jadi 3 setelah sekian bulan. Komunikasi kan jadi lebih lancar sekarang, gak sekacau dulu."
• "[…] In the past, there was only 1 international consumer, now the number of international consumers has increased to 3 after months. Communication is smoother now, not as chaotic as it used to be." (Informant X)

From the measurements at level 4 above, it is known that there is an increase in the number of international partners or consumers as follows: 1) Six (6) participants have 1-2 new international partners or consumers; 2) eight (8) participants have 3–4 new partners or international consumers; 3) three (3) participants have 5–6 new international partners or consumers; and 4) three (3) participants have more than six (6) partners or international consumers.

Discussion

Evaluation of the Four Levels of Kirkpatrick (2006)

This comprehensive analysis section delves into the various facets of the training program, examining its impact across four distinct levels: reaction, learning, behavior, and result. These evaluations are conducted using a comprehensive framework as prescribed by Kirkpatrick (2006), which allows us to assess the overall effectiveness of the program in a structured manner. In the "reaction" segment, it is evident that a significant portion of participants responded favorably to the course material, finding it engaging and valuable. However, within this spectrum of reactions, some participants faced specific challenges. These challenges encompassed issues related to the material's complexity as well as scheduling conflicts arising from their professional commitments. To address the complexity concern

effectively, one prospective strategy involves the implementation of a pre-course placement test. Such a test would serve as a diagnostic tool to gauge the initial English language proficiency of prospective participants. This diagnostic information can then be utilized to tailor the course material, ensuring it aligns with everyone's language capabilities.

Scheduling conflicts, on the other hand, can be addressed through a thorough evaluation process that provides participants with multiple scheduling options. For instance, offering three different schedule slots in a single day could provide participants with the flexibility they need. Additionally, fostering a more serious commitment to adhering to the established schedule is vital to ensuring that skill development aligns with the specific requirements of small and medium-sized enterprises (SMEs). Transitioning to the "learning" aspect, participants, for the most part, expressed satisfaction with the course material, particularly in terms of its positive impact on their oral communication skills. However, a notable concern emerges regarding the limited improvement observed in their English writing skills. This issue necessitates a more balanced curriculum where due attention is given to both writing and speaking components. While substantial progress has been made in oral communication, participants have acknowledged that their written expression skills have not seen the same level of development.

It is imperative to emphasize the significance of proficient written communication, especially within the context of SMEs. This proficiency extends to various domains, including written correspondence, copywriting, contract drafting, negotiations, and the formulation of sales strategies, all of which are conveyed in written form. Consequently, a pedagogical reassessment with a heightened focus on enhancing participants' writing skills is essential. This strategic adjustment is pivotal in equipping participants with a solid foundation for their future business transactions. The "behavior" level reveals that most participants have displayed a noticeable shift in their attitudes, demonstrating a greater inclination to use English not only within the business context but also in their daily lives. However, quantifying this behavioral change exclusively through participant interviews presents a challenge. To ensure that these behavioral changes are consistently applied, it may be advisable for organizational leaders to enforce English language usage as a mandatory practice, thereby fostering an organizational culture that is increasingly comfortable with English. This is consistent with research emphasizing the importance of workplace learning implementation (Dawe & Nguyen, 2007: 35). To provide a more comprehensive evaluation of behavioral changes, participants are encouraged to submit periodic reports, both verbal and written, to the program organizers every three months. These reports should meticulously document changes in their use of English, particularly in terms of its application within the organization. This approach facilitates ongoing monitoring and assessment of the language usage transformation within the organizational context.

In the "result" section, it becomes evident that a subgroup of participants has achieved tangible outcomes as a direct consequence of the training intervention. These outcomes include the acquisition of new customers or the establishment of fruitful business partnerships. This aligns with research findings indicating a positive correlation between internationalization and consumer acquisition (Chelliah et al., 2010). However, these outcomes lack a clear delineation of the antecedent conditions that characterized the interactions among business practitioners. For instance, it is plausible that certain participants had already initiated preliminary engagements with potential customers before the training initiative. Subsequently, the training intervention might have expedited the conversion of these potential customers with greater effectiveness. Conversely, in other cases, the training's effectiveness may have equipped participants with proficient English language skills, enabling them to attract entirely new prospective customers. To gain a deeper understanding of the research outcomes, it is imperative to conduct a more comprehensive exploration of the pre-existing relationships between participants and potential customers. These insights would undoubtedly enhance the research findings, enabling a more precise and nuanced depiction of the impact of the VR training on the participants. By distinguishing the specific contributions of the training intervention within the context of participants' existing interactions and skills, a more accurate interpretation of the results can be achieved.

As we refer to the various discussions, it is evident that the content and utilization of VR technology did not raise significant issues among the participants. Furthermore, within the realm of classroom implementation, technical limitations pertaining to the utilization of VR have not been reported (Bonner & Reinders, 2018). Furthermore, there is a pressing need to comprehensively map out the initial and final conditions of organizational relationships with customers. This mapping process is fundamental for accurately identifying changes resulting from the training program. With this information in hand, a more precise evaluation of the program's impact on participants can be conducted. Addressing these considerations will undoubtedly contribute to refining and tailoring the training program to better align with the specific needs and objectives of both the participants and the organization. This comprehensive approach ensures that the training program remains a dynamic and effective tool for enhancing language skills and cultural integration within the SME context, ultimately fostering growth and success in the global business landscape.

Conclusion

This study focuses on English training programs for small and medium enterprises. The objective was to evaluate the training using the Kirkpatrick evaluation method. Data collection is gathered through FGD, interviews, and standard tests. The collection method is applied according to the Kirkpatrick recommendation. Based on the evaluation at each level, it can be concluded that the training using VR technology is effective. Participants had a positive reaction to the training; they managed to learn the learning material and applied the lessons to their business activities consistently. These changes result in a better business process (especially in terms of external communication) and even market expansion.

The objective of the training for the SMEs relates to the needs of the SMEs' supervisor organization, namely Rumah BUMN Bandung, and must contain the necessary knowledge to encourage the SME actors to be better prepared to become exporters with no obstacles in foreign languages. This way, we can ensure that the training fits the individuals at that point in their development and truly meets their needs to go global. In the early stages, measurements were made at the reaction level. The importance of measuring the reactions of Kirkpatrick (2006) is based on several reasons, namely: to provide valuable input to training providers in improving future training programs; to provide suggestions and input to teachers regarding their level of effectiveness in teaching; to provide quantitative information to decision makers related to the implementation of training programs; and to be able to provide instructors with quantitative information that can be used as a basis for establishing teaching standards for future programs.

At the reaction level, the measurement results show that the satisfaction level of the participants is in the range of 85–90%. This means that the participants show a positive reaction or feel satisfied with the mentoring activities because they realize that they are gaining increased knowledge and abilities. However, there are still some indicators that need to be improved in the training, one of which is the learning materials. Some participants thought that the material provided was too easy, so it was not challenging, while for some other participants, the English material was considered too complicated. This shows that there are weaknesses in the implementation of the training, namely the absence of placement tests for participants based on their basic English skills. Therefore, it is necessary to have a placement test in the next English language training program so that the level of difficulty of the English training material will be set based on the level of language acquisition ability of each participant. Any foundation program must have an English placement test (PT). It aids in placing learners at the appropriate language proficiency level, so they do not waste time learning items that are on the same level as them or one level above them. It helps instructors create lesson plans for students with comparable academic backgrounds. Furthermore, the heterogeneity in responses toward the training regimen emanates from the individualized dedications of the participants; adverse reactions frequently emanate from those individuals whose commitments stand in contradiction to their business responsibilities. This underscores the conceivable requirement to synchronize impending training sessions with predetermined

timetables, thereby amplifying the precision of evaluation through the mitigation of extraneous variables unrelated to the intrinsic instructional content. In addition, what is highlighted related to the satisfaction index towards the English training for the SME actors is the training implementation schedule. It should be highlighted that in Indonesia, the end of the year is the time for SME actors to pursue their business targets.

Learning is regarded as effective if, after the learning process, there is an increase in the knowledge, skills, and attitudes of the training participants. Based on the evaluation theory of Donald L. Kirkpatrick level 2 (2006), to determine the success of a lesson and the level of its effectiveness, data is needed from the test results of the participants, both pre- and post-tests. This stage aims to determine the level of achievement of the learning outcomes. Aspects that are measured in the effectiveness of the learning program cover three domains: knowledge, attitudes, and psychomotor. Without changes in attitudes, knowledge, and skills, training can be said to be a failure. Despite the observed improvements in oral communication, participants acknowledge minimal progress in written skills. However, recognizing the vital role of written proficiency in SMEs, which includes various written aspects of business-like correspondence, copywriting, and contract formulation, calls for a focused evaluation and enhancement of participants' written abilities to ensure their competence in business transactions.

Meanwhile, concerning the aspect of attitudes, the endeavors to influence participants with marginal changes in disposition stimulate the research team to contemplate more frequent training sessions as a means of fortifying the assimilated knowledge. However, a mere escalation in frequency may prove inadequate for effecting substantial shifts in behaviors, considering the weightiness of pragmatic application in the business domain. For an exhaustive examination of these attitudes, supplementary evaluative instruments are imperative, transcending conventional interviews and focus group discussions to encompass the scope of day-to-day business operations. An emerging proposition involves the periodic submission of reports on a monthly basis spanning three months, meticulously elucidating participants' utilization of the English language and the degree of behavioral transformations discernible within their corporate pursuits.

Regarding the final outcomes, certain participants have realized concrete benefits from the training, including obtaining new customers or partners. Nevertheless, these outcomes lack contextual information about previous interactions. For instance, some participants might have engaged with prospects prior to training, potentially augmenting their achievements afterward, while others might have bolstered their English proficiency to attract fresh customers. A more thorough exploration of participants' existing relationships with potential customers would yield enhanced insights, allowing for a more precise evaluation of the impact of the VR training.

Acknowledgement (Optional)

This article is taken from the final report of research conducted in 2022 titled "Komparasi Peningkatan Kemampuan Bahasa Inggris SDM UMKM melalui Metode Konvensional dan Pemanfaatan Teknologi VR", which was funded by Telkom University and Rumah BUMN Bandung.

References

• Al Rawashdeh, A. Z., et al., 2021. Advantages and Disadvantages of Using e-Learning in University Education: Analyzing Students' Perspectives. The Electronic Journal of e-Learning, 19(2), pp. 107- 117, available online at www.ejel.org
• Britt, P. (2004) "E-Learning on the rise: Companies move classroom content online", EContent, Vol. 27, No. 11, pp. 36- 40. Conference Board of Canada. (2001) E-Learning for the workplace: Creating Canada's life-long learners, http://www.conferenceboard.ca/elearning/Downloads/CBoC_SFP/e-Learning_for_the_ workplace. pd

• Bonner, Euan & Reinders, Hayo. (2018). Augmented and Virtual Reality in the Language Classroom: Practical Ideas. 18. 33-53.
• Brunner, Brigitta. (2008). Listening, Communication & Trust: Practitioners' Perspectives of Business/ Organizational Relationships. The Intl. Journal of Listening. 22. 73-82. 10.1080/10904010701808482
• Chelliah, Shankar & Sulaiman, Mohamed & Yusliza, Mohd-Yusoff. (2010). Internationalization and Performance: Small and Medium Enterprises (SMEs) in Malaysia. International Journal of Business and Management. 5. 10.5539/ijbm.v5n6p27
• Coman, Claudiu, Țîru, G. L, Schmitz L.M. Stanciu, C and Bularca, M.C. (2020). "Online Teaching and Learning in Higher Education during the Coronavirus Pandemic: Students' Perspective" Sustainability 12, no. 24: 10367. https://doi.org/10.3390/su122410367
• Dawe, S. and Nguyen, N. (2007) Education and training that meets the needs of small business: A systematic review of the research, Adelaide, Australia: National Centre for Vocational Education Research
• Dahlstrom, Michael. (2014). Using narratives and storytelling to communicate science with nonexpert audiences. Proceedings of the National Academy of Sciences of the United States of America. 111. 10.1073/pnas.1320645111.
• Grzeszczyk, K.B. (2015). Language management in international business. Implementation of strategies to bridge linguistic and cultural barriers.).
• Gilbert, B L (2015)., "Online Learning Revealing the Benefits and Challenges" Education Masters. Paper 303.
• Gherghina, S.C , Botezatu, M A, Hosszu A, and Simionescu, L.N. (2020). "Small and Medium-Sized Enterprises (SMEs): The Engine of Economic Growth through Investments and Innovation" Sustainability 12, no. 1: 347. https://doi.org/10.3390/su12010347
• Haleem, A., Javaid, M., Qadri , M.A., , Suman, R (2022) Understanding the role of digital technologies in education: A review, Sustainable Operations and Computers, Volume 3, https://doi.org/10.1016/j. susoc.2022.05.004. (https://www.sciencedirect.com/science/article/pii/S2666412722000137
• Incelli, Ersilia. (2008). Foreign language management in Lazio SMEs. Language Policy. 7. 99-120. 10.1007/s10993-008-9085-8
• Hanifa, Rizaldy & Yusra, Siti Rahimah. (2018). Insight on Delivering Oral Presentation: Preparations, Problems, and Solutions. International Journal of Learning and Teaching. 4. 318. 10.18178/ ijlt.4.4.318-325.
• Iqbal, M.H., Siddiqie, S A., & Mazid, A.A (2021) . Rethinking theories of lesson plan for effective teaching and learning, Social Sciences & Humanities Open, Volume 4, Issue 1, 100172, ISSN 2590-2911, https://doi.org/10.1016/j.ssaho.2021.100172. (https://www.sciencedirect.com/science/ article/pii/S2590291121000681)
• Kennedy Rolland, F. (2020). The Persuasive Negotiator: Tools and Techniques for Effective Negotiating (1st ed.). Routledge. https://doi.org/10.4324/9781003026884
• Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). San Francisco, CA: Berrett-Koehler Publishers, Inc
• Neuburger, L., & Egger, R. (2017). An afternoon at the museum: Through the lens of augmented reality. In R. Schegg & B. Stangl (Eds.), Information and communication technologies in tourism 2017 (pp. 241–254). Cham: Springer Nature
• Klimek, Sabina. (2009). E-learning for SMEs. Economics and Organization of Enterprise. 3. 71-77. 10.2478/v10061-009-0009-2
• Kostić-Bobanović, M., Novak, M., & Bobanović, M. (2017). The Impact of Foreign Language Skills and Cultural Competencies on SMEs' Success in International Markets
• Lim, H., and F. Kimura. (2010). The Internationalization of Small and Medium Enterprises in Regional and Global Value Chains. ADBI Working Paper 231. Tokyo: Asian Development Bank Institute. Available: http://www.adbi.org/workingpaper/2010/07/29/3972.intl.enterprises.regional.global. value.chains/

• Liu, Ruixue & Wang, Lei & Lei, Jing & Wang, Qiu & Ren, Youqun. (2020). Effects of an immersive virtual reality-based classroom on students' learning performance in science lessons. British Journal of Educational Technology. 51. 2034-2049. 10.1111/bjet.13028.
• Lin, T., & Lan, Y. (2015). Language Learning in Virtual Reality Environments: Past, Present, and Future. J. Educ. Technol. Soc., 18, 486-497.
• Nagro, S. A., Fraser, D. W., & Hooks, S. D. (2019). Lesson Planning With Engagement in Mind: Proactive Classroom Management Strategies for Curriculum Instruction. Intervention in School and Clinic, 54(3), 131–140. https://doi.org/10.1177/1053451218767905
• Neuburger, L., & Egger, R. (2017). An afternoon at the museum: Through the lens of augmented reality. In R. Schegg & B. Stangl (Eds.), Information and communication technologies in tourism 2017 (pp. 241–254). Cham: Springer Nature
• Nurhayati, Y. (2018). Penerapan Model Kirkpatrick untuk Evaluasi Program Diklat Teknis Subtantif Materi Perencanaan Pembelajaran Di Wilayah Kerja Provinsi Kepulauan Riau. Andragogi: Jurnal Diklat Teknis Pendidikan Dan Keagamaan, 6(2), 170–187. https://doi.org/10.36052/andragogi. v6i2.63
• OECD (2019), "Fostering greater SME participation in a globally integrated economy", in Strengthening SMEs and Entrepreneurship for Productivity and Inclusive Growth: OECD 2018 Ministerial Conference on SMEs, OECD Publishing, Paris, https://doi.org/10.1787/400c491d-en.
• Osmičević, Sabina & Meško, Maja. (2020). Professional's Common Skills on Negotiation. Izzivi prihodnosti. 5. 198-212. 10.37886/ip.2020.012.
• Pack, A., & Barrett, A.J. (2021). A Review of Virtual Reality and English for Academic Purposes.
• Palanimally, Y.R. (2016). The Growth Of Small And Medium Enterprises In Malaysia : A Study On Private Limited Companies In Perak Malaysia.
• Parmaxi, Antigoni. (2020). Virtual reality in language learning: a systematic review and implications for research and practice. Interactive Learning Environments. 31. 1-13. 10.1080/10494820.2020.1765392.
• Peleckis, K., Peleckienė, V., Lapinskienė, G., & Dudzeviciute, G. (2016). PRINCIPLES OF EFFECTIVE COMMUNICATION AND PERSUASION IN BUSINESS NEGOTIATIONS. 9th International Scientific Conference "Business and Management 2016." https://doi.org/10.3846/BM.2016.48
• Pérez Fernández, Rodrigo & Alonso, Veronica. (2015). Virtual Reality in a shipbuilding environment. Advances in Engineering Software. 81. 30-40. 10.1016/j.advengsoft.2014.11.001.
• Roy, A. and Raymond, L. (2008) "Meeting the Training Needs of SMEs: is e-Learning a Solution?" The Electronic Journal of eLearning Volume 6 Issue 2, pp 89 - 98 , available online at www.ejel.org
• Schnack, A & Wright, M & Holdershaw, J. (2018). Immersive virtual reality technology in a threedimensional virtual simulated store: Investigating telepresence and usability. Food Research International. 117. 10.1016/j.foodres.2018.01.028
• Tenzer, H., Terjesen, S. & Harzing, AW. (2017). Language in International Business: A Review and Agenda for Future Research. Manag Int Rev 57, 815–854. https://doi.org/10.1007/s11575-017- 0319-x
• Tenzer, Helene & Schuster, Tassilo. (2017). Language Barriers in Different Forms of International Assignments. 10.1057/978-1-137-57406-0_3.
• Ulas, Dilber. (2019). Digital Transformation Process and SMEs. Procedia Computer Science. 158. 662- 671. 10.1016/j.procs.2019.09.101.
• Yee, K. M., Jafre, M., & Abidin, Z. (2014). The Use of Public Speaking in Motivating ESL
• Learners to Overcome Speech Anxiety. International Journal on Studies in English Language and Literature, 2(11), 127–135. Retrieved from www.arcjournals.org
• Yudintseva, Anastassiya. (2023). Virtual reality affordances for oral communication in English as a second language classroom: A literature review. Computers & Education: X Reality. 2. 10.1016/j. cexr.2023.100018.
• Zohar, Ilana. (2015). "The Art of Negotiation" Leadership Skills Required for Negotiation in Time of Crisis. Procedia - Social and Behavioral Sciences. 209. 540-548. 10.1016/j.sbspro.2015.11.285.