4.1. Conceptual Framework

The exploratory nature of this study requires a sound conceptual framework that encompasses, rather than restricts, the multiple dimensions of the issues that need to be examined in a virtual learning environment. Taking into account the multidimensionality of learning as well as virtual reality as a field, a number of technical, orientational, affective, cognitive, pedagogical, and other issues are examined \cite{Lewin95}. The proposed framework, although used specifically for the evaluation of NICE, is meant to serve as prototype for a general evaluation framework that can be used for assessing the educational efficacy of virtual reality experiences.

• Technical
• Orientation
• Affective
• Cognitive
• Pedagogical
• Other (value-added, etc.)

1. Technical

The technical aspect examines usability issues, in terms of interface, physical problems, and system hardware and software.

Interface issues:

a) learning the interface. How long does it take for a child to learn the buttons and joystick of the wand, or the menu? How long and clear do instructions need to be before the child is independent?
b) using the interface. Does the child feel comfort when using the interface device? Does the child use all features and consult the menu? Does the child collaborate with the other children to learn/use the interface?
Physical issues:
a) how many users display motion sickness?
b) how long can the child participate without feeling uncomfortable?
Evaluation of the system hardware and software:
a) robustness for museum/school use.
b) cost/effectiveness for museum/school use
c) flexibility of the program. Can it be expandable and by whom? Can it be authored, modified, customized?
 
2. Orientation

The orientation aspect examines the relationship of the user to the virtual environment, including navigation, spatial orientation, presence and immersion, and feedback issues.

Interface issues:

a) is navigation simple, understandable ?
b) how long does it take the child to understand his/her virtual relationship to the environment?
Spatial orientation:
a) how is left, right, up, and down perceived?
b) how quickly do the children orient themselves?
c) are there any differences between ages or genders?
Presence/immersion: how ``real'' or how three-dimensional does the environment seem to the child?

Feedback: types of feedback (glowing sphere, sound, etc.) and their effect.
 
 

3. Affective

The affective parameter looks at the user's engagement, likes and dislikes, and confidence in the virtual environment.

Motivation: a) is the environment engaging? b) what captivates learners' attention the most? c) for how long? To ensure accurate responses, multiple methods may be employed: observe time spent doing or using something, interrupt children in certain instances based on certain focal points rather than time, ask what five things were liked and disliked about NICE, and use a survey with diagrammatic representations of their feelings (``happy and sad faces'').

Likes and dislikes: a taxonomy of features that the children liked or disliked can be created. For example, likes or dislikes may include the control over the environment, the graphics and animation, the sound, the difficulty in movement, and so on.

Confidence: does the amount of control over the environment, for example, empower the child?
 
 

4. Cognitive

The cognitive aspect identifies any improvement of the subject's internal concepts through this learning experience. Evaluating the cognitive aspect means measuring learning. Cognitive evaluation might start within the environment with the given learning task built into the experience. In NICE, for example, an agent or avatar can give goals to the users or ask them questions (e.g., plant and harvest a row of tomatoes). The responses to these activities could reveal what the user understands about the environment while inside it.

Approaches to the assessment of cognitive abilities include :
qualitative methods of observation

a) users demonstrate their understanding through self explanation with ``think-out-loud'' while doing (can you explain what you are doing?) and ``meta-explanation'' (can you explain what you did/learn?).
b) open-ended written surveys or open-ended oral questions, depending on user's age.
c) continuous video documentation to be viewed and transcribed later.
d) observer as a teacher-avatar in the environment.
quantitative
pre/post tests can be valuable in some situations. In the constructivist approach, the process of knowledge acquisition is equally important to the final product, so evaluation should be embedded in the actual learning process. However, testing skills before and after might also be useful.
 
5. Pedagogical

Teaching approach: how to effectively gain knowledge about the environment and the concepts that are being taught -in this case, ecology or earth science. With respect to virtual reality and NICE, this seems like a second level task once the technical and motivational issues are better known. A collaboration with a teacher who wants to teach something specific would add validity.

Collaboration: do the children learn how to collaborate? How does cooperation aid or hinder learning? Specifically,

a) observe local interaction between 2 or more users (turn-taking, conflict),
b) observe interaction with remote participants,
c) observe differences in communication apprehension and social behavior after the experience.
Narrative : use it as a tool to enhance concept mapping. Does it offer any insights, improve spatial orientation, affect motivation?
 
 
6. Other

``Value added'': what does VR technology add to the instruction that can not be achieved otherwise? This concept could fit into the technical or the pedagogical categories. When children plant a natural garden and then a virtual one, what is it in the virtual that complements the learning experience of the real? Where and how do we measure what the advantages of VR are versus the real thing?

In the case of NICE, for example, what does the ability to shrink and examine the plants, manipulate the environment, or collaborate with children across distance, add to the learning process?