In carrying out the research for this thesis, several studies were conducted, based on a qualitative approach to evaluation. The studies were exploratory in nature (as is the NICE project) and attempted to test out qualitative ideas and issues rather than performance. According to Lincoln and Guba (1985), successful qualitative research is performed through prolonged and persistent observation and through a triangulation of methods, each of which is flawed in some way (cited in Ely, 1991).
Following this model, a wide range of methods for data collection were employed to ensure the assessment of different aspects of this study. These include observation, survey, and interview methods, all of which are interpreted and cross-verified. Several other elements, such as the reports and portfolios produced by the children after the study, were taken into account.
Observation is perhaps one of the most important methods used for the evaluation of VR, as it can provide a first-hand view of the children's behavior and interactions in the virtual world. Both real time observation as well as video and audio recordings were used. All the sessions were videotaped for later observation and transcription. A significant part of the observation is an examination of the kinds of discourse, or ``exploratory talk'', used by the kids during their experience. As a first step, experience from observing NICE users of different ages at various events and conferences helped to refine the methods of observation that would be used for the evaluation of NICE. These users included the attendees of large conferences such as SIGGRAPH '96, computer-literate teenagers at the ThinkQuest event, students ages 12-15 as part of a pre-freshman enrichment program at UIC, and teachers interested in incorporating technology in their instruction at the NCSA Education and Outreach workshop.
The children participating in the main studies were asked to complete written questionnaires before and after their experience in virtual reality. The initial questions attempted to gather as much information as possible about each child's relationship to technology, knowledge of gardening, likes and dislikes. We wanted to establish what knowledge and understanding of the concepts displayed in the environment the children brought with them before the study. After their experience in NICE, the children completed an additional set of questions that related to their impressions and understanding of the environmental relationships in the NICE garden. All of these questions included space for open-ended responses, as they attempted to gather the children's actual thoughts and feelings written in their own words, rather than quantified responses. Additionally, a set of surveys involving the effect of VR on children's level of communication apprehension were conducted by Victoria Pierce and followed the standard pre-test / post-test structure \cite{Pierce97}.
Finally, an open-ended set of interviews was conducted with the children after their experience in virtual reality. The children talked about what they did while in the environment, what they liked or disliked, and what they thought they learned.
The parents were also asked to complete a short questionnaire, pertaining mostly to demographic information -name, age, gender, school information- as well as computer usage information. Both parents and children were asked how much time the kids spend watching television, playing video games, or using a computer per day. This was to avoid biased answers (parents like to think that their children don't spend too much time in front of a screen, so when they say 1 to 2 hours, they really mean 2).
The evaluation included a total of fifty two children: case studies of eight children and two classrooms of twenty two students each. All students volunteered to be part of the study. The ones participating in the case studies were accompanied by their parents, while the classrooms took a school field trip and were accompanied by their teachers. All children returned a parent permission form in compliance with the human subject review panel (IRB \#H-97-078) which is included in Appendix B.
The activities at each evaluation session of NICE took approximately one to three hours to complete, depending upon whether the tests were conducted with groups or pairs of children. This included time to introduce the activity and organize the students, give them time to plan the activity beforehand, perform the activity inside the VR environment, and have some time for post-activity questions and discussion. With the children in the two classrooms there was a familiarization day at their school a few weeks before the activity. This included some pre-test questionnaires to help identify the strength and weaknesses of the children, and their degree of communication apprehension \cite {Pierce97}. The VR setting in all studies included the CAVE and one or two Immersadesks, all linked by an audio connection.
The use of all these techniques should be interpreted as exploratory in nature, involving many qualitative judgments suggested by what came naturally when the research was conducted. The next sections will detail the setting, questionnaires, interviews, and results of each set of studies.