4.1. Conceptual Framework

Evaluation involves gathering and processing feedback on the issues that are being studied with the ultimate goal of demonstrating the worth or validity of a design. The purpose of this framework is to impose a structure on the evaluation of open virtual learning environments, such as NICE. The need for a detailed framework to assess the value of a learning environment follows reasons that have been emphasized in previous chapters (chapters 2 and 3).

The exploratory nature of this study requires a sound conceptual framework that encompasses, rather than restricts, the multiple dimensions of the issues that need to be examined in a virtual learning environment. Taking into account the multidimensionality of learning as well as virtual reality as a field, a number of technical, orientational, affective, cognitive, pedagogical, and other issues are examined \cite{Lewin95}. The proposed framework, although used specifically for the evaluation of NICE, is meant to serve as prototype for a general evaluation framework that can be used for assessing the educational efficacy of virtual reality experiences.

Table I
Framework Category Issue Measurement
Technical Usability Time to learn an interface, comprehension of instructions, physical and emotional comfort
Orientation Navigation, spatial orientation, presence and immersion, and feedback Time to become immersed and comfortable in the environment
Affective Engagement, preference, and confidence Length of engagement, time to reach fatigue, reported and perceived  enjoyment
Cognitive Conceptual change, new skill Performance within and outside the environment, think-aloud and stimulated recall techniques, oral and written surveys, video documentation
Pedagogical Content general and specific teaching techniques Collaboration (e.g., turn-taking, conflict, interaction), avatar acceptance, comparison of techniques
Collaborative VR The added value of collaborative VR to instruction and learning Comparisons of instruction and learning within and outside of collaborative VR environments
  1. Technical
  2. The technical aspect examines usability issues, in terms of interface, physical problems, and system hardware and software.

    Interface issues:

    Physical issues: Evaluation of the system hardware and software:  

  3. Orientation
  4. The orientation aspect examines the relationship of the user to the virtual environment, including navigation, spatial orientation, presence and immersion, and feedback issues.

    Interface issues:

    Spatial orientation: Presence/immersion: how ``real'' or how three-dimensional does the environment seem to the child?

    Feedback: types of feedback (glowing sphere, sound, etc.) and their effect.

  5. Affective
  6. The affective parameter looks at the user's engagement, likes and dislikes, and confidence in the virtual environment.

    Motivation: a) is the environment engaging? b) what captivates learners' attention the most? c) for how long? To ensure accurate responses, multiple methods may be employed: observe time spent doing or using something, interrupt children in certain instances based on certain focal points rather than time, ask what five things were liked and disliked about NICE, and use a survey with diagrammatic representations of their feelings (``happy and sad faces'').

    Likes and dislikes: a taxonomy of features that the children liked or disliked can be created. For example, likes or dislikes may include the control over the environment, the graphics and animation, the sound, the difficulty in movement, and so on.

    Confidence: does the amount of control over the environment, for example, empower the child?

  7. Cognitive
  8. The cognitive aspect identifies any improvement of the subject's internal concepts through this learning experience. Evaluating the cognitive aspect means measuring learning. Cognitive evaluation might start within the environment with the given learning task built into the experience. In NICE, for example, an agent or avatar can give goals to the users or ask them questions (e.g., plant and harvest a row of tomatoes). The responses to these activities could reveal what the user understands about the environment while inside it.

    Approaches to the assessment of cognitive abilities include :
    qualitative methods of observation


  9. Pedagogical
  10. Teaching approach: how to effectively gain knowledge about the environment and the concepts that are being taught -in this case, ecology or earth science. With respect to virtual reality and NICE, this seems like a second level task once the technical and motivational issues are better known. A collaboration with a teacher who wants to teach something specific would add validity.

    Collaboration: do the children learn how to collaborate? How does cooperation aid or hinder learning? Specifically,

    Narrative : use it as a tool to enhance concept mapping. Does it offer any insights, improve spatial orientation, affect motivation?

  11. Other
  12. ``Value added'': what does VR technology add to the instruction that can not be achieved otherwise? This concept could fit into the technical or the pedagogical categories. When children plant a natural garden and then a virtual one, what is it in the virtual that complements the learning experience of the real? Where and how do we measure what the advantages of VR are versus the real thing?

    In the case of NICE, for example, what does the ability to shrink and examine the plants, manipulate the environment, or collaborate with children across distance, add to the learning process?