|Evaluation of an Immersive
Collaborative Virtual Learning Environment for K-12 Education
Maria Roussos and Mark G. Gillingham
851 S. Morgan St., Room 1120
851 S. Morgan St., Room 1120
This paper begins by outlining the current state of virtual reality in education and briefly reviewing a few of the most developed research efforts. The next section presents a short description of the NICE project, the specific virtual learning environment on which evaluation was performed, followed by a structure for the evaluation of immersive VR learning environments. The evaluation methodology section outlines the methods and setting for the experiments. Finally, the next two sections describe the main studies conducted with children and conclude with an analysis of the observed results.
Based on these categories, the virtual reality environments that are designed specifically for education typically fall into three categories. One category includes networked text-based virtual environments, which are highly interactive but not immersive (e.g., MUDs and MOOs, c.f., Bruckman & Resnick, 1995). A second category involves desktop virtual reality simulations (e.g., walkthrough applications, virtual classrooms; c.f., Winn, 1993; Cromby, Standen, & Brown, 1995; and Bricken, 1991), where interactivity is usually limited but varies according to the control given by the program, and immersion also varies but is not easily provided. A third category includes the immersive VR environments, where immersion is high, but interactivity may be limited, depending on the complexity of the virtual world (e.g., head-mounted display, CAVE, c.f., Dede, Salzman, & Loftin, 1996; Bricken, 1991; Rose, 1995; Winn, 1993; Cruz-Neira, Sandin, & DeFanti, 1993; Gay& Greschler, 1994). We provide a more detailed description of these three categories of virtual realtity (Roussos, et al., 1997).
Our interests lie in the evaluation of the highly visual and interactive VR environment developed and discussed later in this work. Thus, throughout this paper, virtual reality will refer to immersive virtual reality and not the other kinds, unless noted.
The NICE garden was originally designed as an environment for young children to learn about the effects of sunlight and rainfall on plants, the "spontaneous" growth of weeds, the ability to recycle dead vegetation and similar simple biological concepts that consist the life cycle of a garden. Since these concepts can be experienced by most children in a real garden, the NICE garden provides its users with tools that allow its exploration from multiple different perspectives: in addition to planting, growing, and picking vegetables and flowers, the children have the ability to shrink down to observe the root system of their plants or meet other underground dwellers, leap high up in the air, climb over objects, and experience firsthand the effects of sunlight and rainfall by controlling the environmental variables. Familiar methods of interaction are employed, which eliminate the use of menus and instead use simple visual metaphors. The children can thus water their plants by pulling a raincloud over them, provide sunlight with the use of the sun, or clear the garden weeds by recycling them in the compost heap. The symbolic representations of the various environmental elements as well as instant feedback are used to facilitate the learner's understanding of the biological relationships which take place in the garden. Thus, when the raincloud has been over a plant for too long, the plant holds an umbrella; when it's too sunny, it wears sunglasses.
In addition to facilitating social interaction amongst the children working on NICE in the CAVE, collaboration is also enabled across virtual communities of children, visually and through real-time remote audio. The network component of NICE allows multiple networked participants to interact simultaneously with the same virtual environment and each other. Each remote user's presence in the virtual space is established using an avatar -a graphical representation of the person's body in the virtual world. The avatars have a separate head, body, and hand, which correspond to the user's actual tracked head and hand motions. This allows the environment to record and transmit sufficiently detailed gestures between the participants, such as the nodding of their heads, the waving of their hand, and the exchange of objects.
Additionally, students without access to a VR system like the CAVE, can still participate through any computer connected to the Internet. With the use of a simple JAVA interface, children interacting with a two-dimensional version of NICE on the Internet can simultaneously share and manipulate the same three-dimensional space as the children in the CAVE. They can also converse with all other, virtual and remote, participants by typing in the provided text area -a feature which resembles text-based virtual environments, such as MUDs. To extend the activities beyond the computer even further, every action in the environment and every interaction amongst the virtual participants adds to the story that is being formed continuously, as the virtual world never ceases to evolve. The action is captured through a simple transcript and automatically parsed to look like a picture book, which is then placed on a public WWW site. The children can access and print their garden and their stories anytime (Steiner, 1994).
A detailed discussion of the objectives, the ongoing design process, and the complex architecture of the NICE project can be found in other publications (Johnson, et al., 1998, Roussos, et al., 1994, Roussos, et al., 1997).
A conceptual framework was developed based on these directions. The purpose of this framework is to impose a structure on the evaluation of open virtual learning environments, such as NICE. The exploratory nature of this study requires a sound conceptual framework that encompasses, rather than restricts, the multiple dimensions of the issues that need to be examined in a virtual learning environment. Taking into account the multidimensionality of learning as well as virtual reality as a field, a number of technical, orientational, affective, cognitive, pedagogical, and other issues are examined (Lewin, 1995) and skecthed out below.
|Technical||Usability||Time to learn an interface, comprehension of instructions, physical and emotional comfort|
|Orientation||Navigation, spatial orientation, presence and immersion, and feedback||Time to become immersed and comfortable in the environment|
|Affective||Engagement, preference, and confidence||Length of engagement, time to reach fatigue, reported and perceived enjoyment|
|Cognitive||Conceptual change, new skill||Performace within and outside the environment, think-aloud and stimulated recall techniques, oral and written surveys, video documentation|
|Pedagogical||Content general and specific teaching techniques||Collaboration (e.g., turn-taking, conflict, interaction), avatar acceptance, comparison of techniques|
|Collaborative VR||The added value of collaborative VR to instruction and learning||Comparisons of instruction and learning within and outside of collaborative VR environments|
The main study sessions were conducted using a total of 52 children: 44 second-grade children from an urban Catholic elementary school with an ethnically mixed student population; another 8 children from other schools participated in case studies after the classroom studies were completed. The gender distribution was equal: 26 boys and 26 girls. The activities at each evaluation session of NICE took approximately one to three hours to complete, depending upon whether the tests were conducted with groups or pairs of children. This included time to introduce the activity and organize the students, give them time to plan the activity beforehand, perform the activity inside the VR environment, and have some time for post-activity questions and discussion. The VR setting in all studies included the CAVE and one or two Immersadesks, all linked by an audio connection.
The teachers were asked to evaluate the students in their class according to their reading and writing skills, leadership skills, and shyness. The children were then randomly assigned to groups. We tried to keep the groups as equally distributed as possible by selectively matching and assigning the children with strong leadership skills or strong reading and writing skills to different groups. Each class of 22 students was divided into three teams of 7 to 8 students each.
Before beginning the VR experience, the children were asked to complete pretest question sheets. These initial questions attempted to to identify each child's relationship to technology, familiarity with gardening, and understanding of simple ecological concepts. We wanted to establish what knowledge and understanding of the concepts displayed in the environment the children brought with them before the study.
After completing the questionnaires, each group of students was asked to generate ideas for planning their garden. A large piece of paper containing a top-down view of the garden was given to each group. Four rows of differently colored stickers, each one representing one of the four available vegetables, were provided. The children in each group had to plan where they would plant their vegetables by placing the stickers on the soil area of the garden. A total of forty vegetables were allowed (10 stickers for each kind).
After the planning stage, the first team continued onto the CAVE and ImmersaDesk part, while the other teams remained in the room to continue their concept maps. Each team was split into two groups, one for the CAVE and the other for the Immersadesk. The two groups collaborated remotely, represented by the avatar of the leader of each group. The leader was assigned randomly by the researchers, to avoid conflicts during the experience in VR. The leaders were instructed in the use of the wand and were allowed a 10-minute period to practice navigation. Each session lasted for an average of 30 minutes. In addition to the two avatars sharing the same virtual space, an adult acting as teacher was disguised as a girl avatar and was guiding the groups from another Immersadesk. The teacher-avatar was also responsible for keeping the time, keeping the children focused on their planting task, helping them accomplish the the planting planned on paper, and encouraging the two groups to speak louder. An audio connection between the three VR sites was established through the use of hidden ambient microphones. Out of a total of 8 groups for each classroom, 4 groups were of single gender (2 all-girls teams, 2 all-boys teams), and the remaining 2 were of mixed gender.
Following the virtual experience, an open-ended set of interviews was conducted with the children where they completed an additional set of questions that related to their impressions and understanding of the environmental relationships in the NICE garden. The questions included space for open-ended responses and discussion with the researcher, regarding what the children did while in the environment, what they liked or disliked, and what they thought they learned.
After the interview, the groups returned to the main room. Large pieces of white paper were placed on the table, upon which the students could draw. They were asked to draw the garden they created in NICE. The activities also continued in their classroom after the experiments. The teachers assigned to the students homework reports to describe the virtual reality experience and propose their own virtual worlds.
Similar methods were used in the case studies, but the children were in pairs, rather than groups, and the time they spent in the virtual environment was longer. One boy who participated in the initial study returned two more times, where he collaborated once with a remote adult and next with another remote boy of his own age.
As with the initial study, the children in the main studies exhibited diversity in their use of the wand, the interaction device. The instructions given were not exactly the same for everyone but depended on the situation and the environmental or personal distractions. Generally, these instructions started with showing the representation of the virtual hand to the leader, then the use of the joystick for navigation, and finally, once the child was able to move into the garden, the function of the buttons. Learning the functions of the wand lasted from 2 minutes, for the children that learned quickly, to 7 minutes.
After learning how to use the wand, the children's effort was focused on orientation, as noted in the following section. Limitations of the physical design of the wand caused discomfort to young users, as both hands were needed to reach the buttons and press the joystick at the same time. The joystick of the current wand is difficult even for adult users and requires applying considerable force when navigating. The children obviously had a problem doing this. It was expected that the boys would generally be better at using the wand, partly because of their strength and partly because of their familiarity with similar input devices from playing video and computer games. According to both parents' and kids' reports, 92% of the boys play electronic games weekly, as opposed to 42% of the girls. The majority of these games have joystick-based interface devices. We did not notice, however, any gender differences in learning to use the wand.
A larger problem was the size of the stereo glasses. Despite the glass-ties used to tighten the glasses on the children's heads, the glasses would still fall off. Most children had to hold the glasses with their free hand and, when tired holding them, would just take them off. Not only did this contribute to the subjects' fatigue, but also to their level of motivation and excitement. Since the stereo glasses and the wand are an integral part of the virtual experience, these limitations are a current hindrance not only to usability but also to learning.
Some of the best "drivers'' consulted the menu that was attached to the virtual hand. These instances, however, were very few and sometimes inconsistent: some of the drivers who consulted the menu at the beginning forgot about it later, while others remembered to use it only part of the time.
The children's susceptibility to simulator sickness was not as large as expected. Less than 5% of the subjects complained about getting a headache or being dizzy during or after the experience, and for most it was so slight that they had not noticed until asked. Only a few girls, mainly ImmersaDesk users, felt dizzy during the experience and for longer afterwards. One girl felt slight nausea commencing about 15 minutes into the experience, and lasting for about a half hour.
Evaluation of the system with respect to its robustness and cost effectiveness for broader use must be put off until the system is in a public locale. Nevertheless, it is necessary to mention the large number of human staffing required for these studies. Approximately eleven people helped each day of the classroom evaluations. Part of this number relates to the evaluation procedure itself (videocamera person, guides, interviewer, etc.), but a core number of at least three people, including a technical person, an instructor to handle the glasses and teach the use of the interface devices, and a teacher-avatar are needed for even the simplest case study. The NICE software is flexible enough to eventually expand into a user-authoring system. To be effective, however, it needs to be used by a small number of learners for an extended period of time.
|Computer use for school||4.14||2.4|
|Parent computer usage||5.85||6.16|
After learning how to use the wand, the children focused on trying to navigate and orient themselves in the virtual environment. With respect to the classroom groups, this proved to be the effort of the leader and not of the other children in the group, although their mission was to help the leader. The drivers were the only ones focused on the orientation task at hand, as they were the ones navigating, while the other children were distracted by the movement and the three-dimensional graphics. The girls seemed slightly better at orienting themselves in the environment, possibly because they were generally more focused and reserved compared to the boys. Even with the case studies, although not nearly to the same extent, there were times when the other child (the one not using the wand) would wander around, instead of observing or directing the driver's actions. While it was not expected that all children's full attention would be given at orientation, the result in these studies was that each child came up with their own version of the right direction, voiced them at the same time as the other children and confused the leader, who then individually decided which was the right path to take. As a result, apart from the difficulty in using the joystick for navigation, the leaders exhibited noticeable individual differences in their abilities to interact with the 3-D environment. These differences seemed to relate to their level of "independence:" the ones pursuing their own goals did well, while the ones that attempted to listen to the others in their group ended up confused and disoriented.
A test for spatial orientation was the ability to find areas in the space, such as the hole that leads to the area under the garden. This was a relatively hard task, although there were spatial clues: the passage was located near the only set of trees behind one of the garden fences. These were the only instances where verbal interaction seemed to work well, largely because the goal was very specific and required the kids' complete attention.
Another test for orientation was the concept map -the plan of the garden on paper. In the planning stage, students developed different strategies for planting. We wanted to see how they were able to implement this plan in VR. The case studies were more focused and, therefore, the children attempted to stick to their plan. With the exception of the boy in the initial study, the children were not successful at completing the task. Most children began planting as planned, but then changed their plans when running into difficulty. A younger girl who tried following the plan, commented that it was very hard to be precise in separating the vegetables. The teacher-avatar helped her with directions, but that "wasn't enough."." The classrooms, on the other hand, hardly even tried to implement the plan, although constantly reminded by the teacher-avatar. Their entire experience was consumed by dealing with the group's behavior. None of the children admitted that they did not try; rather they stated that implementing the plan was a difficult task. One boy, after seeing the look of the group's final version of the garden asked his group: "how come we didn't get it right?'' to receive the overwhelming response "because it was very hard!"
As perceived through observation, most kids felt immersed. This was indicated by their motion and excitement. Almost all children attempted to "touch'' the virtual objects by moving and clasping their hands in the air. This was particularly noticable in the case of the virtual beam that extended from the user's hand to help point to and select objects. As the beam was always attached to the hand and close to the user, it felt very "three-dimensional'' to the children. Many children, however, would take their stereo glasses off and put them back on constantly during the experience. One of the boys in the case studies would take the stereoglasses off every time he needed to accomplish a more demanding task, such as finding his way underground. When asked why, he answered that it was easier to see without the glasses because "they were heavy and I couldn't do things right." Technical obstacles, such as the size of the stereo glasses, possibly hindered the perception of presence.
The present feedback seems effective, as most children understood the function of the yellow balloon for picking, and the virtual hand. Many leaders waved at the other avatars with the hand that was holding the wand, indicating that they understood the relationship between the wand, their real hand, and the virtual hand.
Measuring motivation is difficult, as it is indirect. We do not see motivation, but behavior (Anderson, Ball, & Murphy, 1975). Moreover, in the case of virtual reality, motivation is highly driven by other factors, such as the novelty effect, media hype, and social issues. It is significant to look through these factors and try to identify whether the content taught within this medium is motivating for children, what it is that motivates them, and most importantly, for how long. This was difficult, as all of the children were excited before starting, just by the fact that they would experience virtual reality. Therefore, we had to look at their level of engagement during the actual experience.
The amount of time the children spent in VR ranged from 30 minutes to 1.5 hours. Each classroom group, due to time constraints, stayed in the experience for about 30 minutes. The case-study subjects, on the other hand, were allowed to stay until they displayed noticeable fatigue, at which point they were asked if they wished to continue. Most cases wished to remain in NICE for at least 45 minutes and started getting tired after one to one and a half hours.
Interactive activities ranked high amongst the preferences of the children, as shown by their responses in the post study questions. Planting was a favorite. An equal number of responses were in favor of the area under the garden. The fantasy was another fundamental driving force for many of the children. Many liked the water (or "swimming''), the rain, sun, umbrellas and sunglasses, and the vegetables. The three things that were most disliked by the children included "the stuff that we had to move with," the "glasses falling off," and the fact that some did not get to drive. Most (73%) of the children answered "nothing'' to the question "what did you dislike the most?."
The most important issue related to motivation is control. As mentioned in the discussion of orientation, the children that were leading were more on-task and engaged, while all others were distracted and unfocused. This was also perceived, to a lesser degree, with the pairs of children in the case studies: the driver was focused on the task even if that meant only navigation, and was consequently more engaged, while the second child seemed less engaged. The post-experience questions verify these observations: Children that were leaders listed that what they enjoyed the most was being the leader, while most others that did not get that chance were very disappointed.
Examining the cognitive value of a virtual learning environment is very difficult, as there are many other factors which correlate to learning, such as the ones described above. Particularly, distraction, fatigue, and cognitive overhead in mastering the interface influence the outcome. The classroom studies provide good examples of a situation in which all the above took place, and where one cannot derive any conclusions about conceptual learning. The results from the case studies are more promising, as the studies were more focused, prolonged, and with less noise and disorder.
However, even in the case studies, little can be concluded as far as learning is concerned. Confidence in using the interface does not necessarily signify understanding of the subject matter. One of the boys, for example, who reported playing many hours of video games per week, learned the interface very quickly and easily and had very good navigation and picking skills. After interacting with VR for about 40 minutes he was interviewed. During the interview and his post-study questions it was revealed that he had not perceived the effects of the sun and the rain on the plants, nor the function of the umbrellas and sunglasses. This was consistent with his pre-study test, which showed little knowledge of gardening concepts.
To simplify the understanding of the children's knowledge before and after the virtual experience, their responses were grouped into categories. For the pre-study test, three categories were devised according to the children's understanding of simple ecological relationships. The first category included the responses that displayed a very good understanding of gardening concepts: the plants need water and sunlight (i.e. good temperature), and good soil to grow, they wilt or look brown when they are sick, they wilt if they get too much water and dry out when they get too much sun, and the weeds need to be pulled out. About 12% of the subjects answered in this way. They were also the ones ranked high in reading/writing skills by the teachers. The second category included most of the above answers except for a few misconceptions (e.g. water is good but sunlight is bad for plants). 42% of the children's answers fit into this category. The third category included 44% of the responses, where more than one question included a "don't know'' response or a wrong answer (such as "the plants grow down'' when they get sunlight, or that weeds need to be planted and watered). Finally, one child could not answer most of the questions.
The answers to the post-study questions were grouped into categories based on the children's understanding of the NICE model: the plants display umbrellas when they receive too much water and sunglasses when there's too much sun, while the weeds are recycled in the compost heap. The responses here were more difficult to categorize, as many children had trouble synthesizing their learning during post-testing, due to fatigue or excitement, while others misunderstood the questions and answered in the same way as in the pre-test, not understanding that the post-questions pertained to the NICE garden in particular.
Approximately 17 children (35%) understood, for the most part, the NICE model. Of these 17, 13 were drivers, and all had done well in their pre-study questions. This shows that most of the leaders, children that were actively engaged in the task, understood the model of the NICE garden, whereas only a few of the other children perceived it. Approximately 45% of the children simply answered "they grew'' to the questions "what happenned when you put the rain over the plants'' and "what happened when you put the sun over the plants." Five kids answered that they did not know or see what happened while six kids were tired and did not answer at all.
The children acted naturally while in NICE, just as they would have at a playground. They played, argued, listened, spoke loudly, and even rested. Very few were curious about the technology, excepting a girl who exclaimed that the screens were made of paper. The presence of "the computer'' was not generally perceived by the children throughout the sessions. As one child put it, "I thought we were going to play with a computer, but this was different." This indicates that perhaps virtual reality can provide a natural medium for teaching, once technical and technology-specific problems are resolved.
Although children in these studies participated in the VR session longer than in any other educational VR study, it appears that this was not an important factor in the facilitation of learning. We must agree, however, with Dede (1996) who reports that spreading lessons over multiple VR sessions appears to be more effective than covering many topics in a single session, as we attempted to do in our studies. Reviews and post-tests from their studies demonstrated that students were better able to retain and integrate information over multiple lessons. This is usually the case in school-based learning as well as being the main concept of life-long learning.
With respect to their pedagogical function in the NICE studies, collaboration and the narrative are explored further in the following sections.
The classroom studies were set up to encourage intra-group collaboration and inter-group competition, to ensure that each group had an incentive to focus on the task of creating a tended garden. However, none of these forms of cooperation occured. After each group was split, one sybgroup to go to the CAVE and the other to the ImmersaDesk, the children had to be continuously reminded by the teacher-- avatar that they were still one group working on a common goal in the same garden. Most children, however, continued not to perceive this and regarded the other (remote) half of their group as their competitors. There were multiple instances of the two drivers fighting over who would grab the raincloud, and children from one location yelling at the ones in the other location to step out of "their'' garden. As far as the classrooms were concerned, competition contributed to the excitement of the children in the group, but kept them off-task and distracted them for nearly the entirety of the experience. Some of the groups even displayed a form of intra-group competition between the leader and other members. This related mainly to the control of the wand. Notable is the case of one girl who caused constant conflict because she was not the one chosen to be in control. The intent during these studies was to have only one child in each group control the wand. Our rationale for this was efficiency: it is easier and quicker to teach one subject than all; it is more efficient for one to control while others direct the activity, and it avoids fighting over who will do it.
On the other hand, this efficiency gain might not be helpful in terms of advancing all the students' learning. In the case of the other students, it was evident that the control over their learning and their experience was in the hands of the leader of the group. It was hoped that, in this way, the students would be able to pay more attention to the subject matter by leaving the control of the learning situation to the leader. For the child controlling, we supposed that this would not be an advantage, as it could lead to less attention to the subject matter and more to the task of controlling. As noted previously, the opposite was observed in these studies: the leader paid more attention to the subject than the other, less active members of the group.
Contrary to the classroom's behavior, the pairs of children in the case studies displayed excellent collaboration and no competition. In most cases, on-task communication was observed and there was general agreement on actions. Based on these observations, issues regarding the selection and number of members in a group must be taken into account for a successful collaborative combination.
For both the classroom as well as the case studies, the teacher-avatar seemed to serve a helpful purpose, especially for giving the kids tips and keeping them on task. In terms of the classroom children, of course, the teacher-avatar consumed most of her time attempting to keep order - not unlike a real classroom.
The system's visual output (the narrative WWW page) was shown to each group during the interview. Each group was represented in the story by the avatar of the leader. Some children did not understand this until it was explained to them while showing them the narrative. Most were fascinated by the pictorial representations of the characters and vegetables and remembered what they were doing by looking at the story. It is believed that the iconic representation was helpful in giving the groups a general overview of their actions and is worthy of further exploration. An unanticipated function of the story was its use as a spelling aid by two children from different groups. When completing their questions, they asked to look at the WWW page to find the spelling of certain object names.
Several problems experienced in these studies point to issues that must be addressed to make adequate conclusions with respect to learning. Specifically, group dynamics, in the case of the larger classroom groups, overcame the effects of the virtual reality experience in several instances. Instead of collaborating, the groups in different physical locations seemed to conflict. Also, subdynamics within each group took place; the other members often hindered the leader if he or she paid attention to them at all. The issue of control over the environment, in some cases, became more important to the group members than their experience. We can conclude from this that either larger groups don't work at all in virtual environments, or more likely, that larger groups need to be more carefully orchestrated in order to achieve effective results. These group factors can be more powerful than the experience itself, and therefore in the design of any virtual learning environment careful decisions about the size and selection of the group, as well as its social aspects, should be made.
Perhaps the most important conclusion these studies point toward is the role VR can play in the actual quality of learning. The affective factor is one of the most important characteristics surrounding VR in general, encompassing interaction and engagement, but in these studies was not limited only to increased motivation in the children. The amount of interactivity and engagement directly influenced the outcomes related to the effectiveness of the learning process. Learning was directly tied to the children's level of engagement: the ones actively involved were more engaged and motivated to complete a task and consequently understood the model employed by the NICE garden. The leaders (the ones actively using the control device to drive the interaction) got the most out of the experience while passive viewers did not understand the concepts presented as well as the leaders. This is a tentative conclusion, since the children were chosen as leaders based in part upon their cognitive skills (leadership ability, cognitive skills, and height, in that order). Nonetheless, further study in this area is promising.
These studies provide preliminary observations in the area of virtual reality evaluation. Future work includes smaller studies with multiple cases, directed towards the exploration of cognitive and pedagogical issues. As virtual reality systems are becoming more accessible to the public, issues in the design of virtual learning environments must be addressed. We think that the development of research-based environments such as NICE, coupled with concrete evidence of their educational value, provide an essential step in this direction. In addition, we think the conceptual framework used to guide this study will help determine the place of VR in educational settings as more study adds detail to the framework.
To read more about the NICE project or participate in the experience through the JAVA interface, please visit: http://www.ice.eecs.uic.edu/~nice/