Initial and main studies were performed as part of the evaluation of NICE. Both these studies are described in the following setions.
4.3.1. Initial Study
Pilot studies are used as precursors to larger, more formal evaluation studies. The initial study conducted for NICE was limited in comparison to usual pilot studies. Two case studies took place about a month apart from each other; the first was the study of a 9-year old boy, while the second included two girls, one 9 and the other 7 years old.
The research objectives of these studies were twofold: First, to examine issues regarding the usability of the system, to become familiar with the procedure for conducting studies in the CAVE, to solve any technical problems early on, and to gain experience, so as to refine the testing process for the main studies. Second, to collect some preliminary observations on the differences in the childrens' interaction with the environment. We wanted to see if the differences in the habits, behavior, and attitude of the children affect their ability to interact in the virtual environment. The observation focused on what questions were asked by the kids, what they liked or disliked the most, where they kept their focus, how well they understood the goal. An important point in the design of the study was the focus on the differences between children that play video games and the ones that do not. In this sense, the initial studies were intended as comparative studies. 220.127.116.11. Method
The subjects were selected according to gender and level of computer literacy. Both selection criteria are related, as differences with respect to the use of technology have been reported between boys and girls. This is mostly due to stereotypes and the focus of the video game market on boys. Consequently, boys start using technology earlier and are more comfortable and confident with it than girls. We wanted to see if this behavior extends to VR. The boy selected is an experienced video game player and computer user, whereas the two girls spend most of their time reading books and hardly ever use a computer. None of the children had previous experience with the NICE environment. The boy, however, had experienced a commercial entertainment-based immersive virtual reality environment.
The CAVE and one Immersadesk were used for the initial studies. The boy was studied interacting alone in the CAVE for 5 minutes and then with a remote character, controlled by the researcher, for the remaining 65 minutes. The two girls were sisters and preferred to start out in the CAVE together. An avatar appeared after 30 minutes, and then another 30 minutes later the girls split and interacted with each other remotely. The younger one remained in the CAVE while the older continued from the Immersadesk. Their virtual reality experience lasted a total of 1 hour and 30 minutes. No audio connection had been set up for these initial studies, so the avatars communicated only with gestures.
After the experience the children were interviewed. Additionally, a set of pre and post study questions were asked, to identify communication apprehension problems.
Noticeable differences between the two case studies were observed during this first use of the CAVE environment with children. The boy, an avid video game player, learned how to use the interface very quickly and with hardly any instruction. His movement in the environment was confident, displaying complete immersion. He was very dynamic and animated, using his full body and turning his head a lot. Many times he kneeled or sat on the CAVE floor when the task required it. He accomplished his tasks in a stable and precise manner steadily picking and planting vegetables with a clear pre-planned sense of organization. He asked and answered his questions on his own but had to be reminded to speak ``out loud''. The boy's favorite part was the interaction with the remote character. He waved and followed the girl-avatar that was controlled by the researcher, or tried to keep the bird-avatar, which was trying to steal plants, out of the garden by moving his hands around and commenting out loud that he wished he had a scarecrow.
The girls, on the other hand, although engaged, stood fairly still when using the wand and faced straight ahead when moving around. They were not as talkative or comfortable at the beginning. Towards the end they were considerably more talkative, but did not seem to get better at navigating or feeling more confident when moving around in the environment. As the older girl noted, ``you want to go one way and it moves the other way''. Their collaboration, however, was natural: they switched glasses often to give each other a chance, or to help solve a problem. While the boy was much more adept at picking up objects, even those that were moving, the girls were not. Indicative is the older girl's comment about planting a vegetable: ``sometimes when you want to plant it doesn't get yellow''. She had not perceived that the yellow bounding sphere, indicating that the object may be picked or placed, would appear when pointing at the soil. The boy had understood this feedback cue right away. The girls' sense of organization was also very different from the boy's, indicated by the way they planted their garden. Their decisions lacked a clear structured manner and were fairly random. Their favorite part was the underground. They were both more animated and motivated when searching for the passage to the underground. The most notable difference was their behavior towards the other avatar. Unlike the boy, they almost completely ignored the researcher-controlled boy-avatar who entered the environment and was trying to get their attention. This doesn't mean that they didn't understand what he was doing, because while he was trying to get their attention, we asked one of the girls what the character was doing. She replied: ``He's telling me to come here," and continued doing what she was previously doing without paying attention to the character.
18.104.22.168. Identified Problems
These early tests pointed out or reconfirmed some problems regarding the user interface of the VR system. First of all, the size of the stereo glasses turned out to be a problem. The use of glass ``ties'' to tighten the large glasses on the head did not help in the case of the girls because they were switching the pair of tracker glasses constantly; each one wanted to make sure that the other had the same shared experience. Unfortunately, no better solution can be found for this problem until the industry produces new glasses.
Another problem, related mainly to testing, was the lack of an audio connection between the two VR systems. NICE's environmental sounds overpower the children's voices in the CAVE, making the extraction of information from the recorded video source very difficult. The addition of a wireless microphone is a solution for louder and clearer sound, but adds another piece of equipment for the kids to wear. This problem was solved in the final studies by using an ambient sound connection.
An attempt to have the children demonstrate their understanding during the experience, by thinking out loud, did not produce the desired results, as the children were a little shy and spoke softly. In the formal studies we used a ``teacher avatar'' to overcome this problem.
Some technical problems which occurred during both tests, such as the failure of the logging program to record the time, were corrected in time for the main studies.
In terms of the experiment design, the main problem was that of too many differences between the children: this study confirmed that a comparison between the children was difficult and its validity questionable. While there was an attempt to provide almost identical testing conditions, so the studies would be smoother and the comparison more valid, there were three major difference factors between the two case studies: gender, computer literacy, and the fact that the boy was alone, whereas the girls were together in the CAVE. Moreover, differences between the individuals in a comparison study of this size may overwhelm other factors.