Evaluation Techniques and Re/Inducing Sickness

(material from: Information Anxiety by Saul Wurman, The Psychology of Human-Computer Interaction by Stuart Card and friends, Designing the User Interface 3rd Ed. by Ben Schneiderman, Human-Computer Interaction 2nd by Dix etc)

3 goals

• assess the extent of the system's functionality
• assess the effect of the interface on the user
• identify any specific problems with the system

Kenneth Boulding "The moral of evolution is that nothing fails like success because successful adaption leads to the loss of adaptability ... This is why a purely technical evaluation can be disastrous. It trains people only in thinking of things that have been thought of and this will eventually lead to disaster"

How evaluation is done depends on many factors

• stage of design
• novelty of project
• number of expected users
• criticality of the user interface
• costs of product and finances allocated for testing
• time available
• experience of the design and evaluation team
  
  

Different types of Evaluation

Evaluating the Design

   cognitive walkthrough

• detailed review of a sequence of actions
• main focus is on how easy the system is to learn for a new user
• given:
• description of system prototype
• description of task user is to perform
• complete written list of actions user must perform
• indication who the users are and what experience they have

    heuristic evaluation

• multiple evaluators
• main focus is evaluating early designs
• 10 heuristics
• visibility of system status
• match between system and real world
• user control and freedom
• consistency and standards
• error prevention
• recognition rather than recall
• flexibility and efficiency of use
• aesthetic and minimalist design
• help users recognize, diagnose, recover from errors
• help and documentation

    review-based evaluation
            look through existing literature for previous related experiments

    model-based evaluation
            GOMS, keystroke-level model, etc

Evaluating the Implementation

    qualitative
    quantitative

    Expert reviews
  

Usability Testing

• in the laboratory - controlled but may be unrealistic and short term focus
• in the field - longer term, more realistic but harder to control

• informal testing with mock ups
• thinking aloud
• video and audio recording
  

• tends to emphasize first time usage and limited number of features
• pilot studies are very important to find errors in the testing procedure
  
• run through the entire experiment with a small group of subjects
  
• participation should be voluntary and FULLY informed
  
• user should feel they are not being tested
  
• important to collect data about the participant's background
  
• privacy of records is very important
  

    Surveys

• Questionnaire for User Interaction Satisfaction (QUIS)  http://lap.umd.edu/quis/
• System Usability Scale (SUS) http://www.measuringusability.com/sus.php
• Computer System Usability Questionnaire (CSUQ) http://hcibib.org/perlman/question.cgi
  

    Acceptance Tests
         establish specific testable criteria for the application:
            time to learn, speed of usage, rate of errors

Controlled Experiments

• come up with a hypothesis that is testable and measurable
• set up an experiment where certain control variables are varied
  

• subjects
• match expected users
• should have at least 10 subjects, in general more is better, 50-60 not uncommon
  
• thoughts from Jakob Nielsen - http://www.useit.com/alertbox/20000319.html
• usually you pay them for participating
  

• variables
• variables that are manipulated - independent variables 
• each independent variable can have a number of different values - levels
  
• variables that are measured - dependent variables
  
• manipulate independent variables to produce different conditions for comparison
  
• dependent variables should be only affected by the independent variables

• hypothesis
• prediction that varying the independent variables will affect the dependent variables in a certain way
  
• goal is to show that this prediction is correct
  
• disprove the null hypothesis (no difference in the dependent variable between levels of independent variable)
  
• produce values to compare to various levels of significance
  
• if its significant, at some level of certainty, that differences would not have occurred by chance

• experimental method
• between groups (randomized) - each subject assigned to a different condition
  
• each user only does 1 condition
  
• experimental condition - the variable has been manipulated
• control - experimental condition without manipulating the variable
  
• need more subjects
  
• differences among subjects can bias the results
• within groups
  
• each user performs under each condition
  
• possible problems with transfer of learning effects
  
• need fewer users

• statistics
• LOOK at the data and SAVE the data

We can also get a lot of information using eye-tracking systems telling us where people are actually looking in addition to mouse clicks and touches telling us where they are interacting.

http://www.uxmatters.com/MT/archives/000040.php

and eye tracking is getting cheaper and more available with gadgets like the Tobii

and also more and more data tracking users positions, gestures, etc.

Video and Audio recording, especially when a subject is speaking aloud, can be incredibly useful for finding interesting subtle things, but it can take a very long time to go through. Early on in evaluation you are often looking for interesting events or sequences of events. In a more refined study you have a set of predefined codes for particular events and go through the video and code the time in the video you saw each particular event that you are interested in.

IRB (Institutional Review Board) issues
http://tigger.uic.edu/depts/ovcr/research/protocolreview/irb/contact.shtml

and especially issues of informed consent
https://en.wikipedia.org/wiki/Informed_consent

here is a consent form and media consent form that we used last year

Send andy an email or a link to a web page discussing how you would study the usability of your mirror.

•     What are the three most important things you would have your pool of users evaluate?
•     What kinds of users would you recruit? and how would you recruit them?
  
•     How would you evaluate the mirror in a laboratory condition? What data would you collect? Over what period?
•     How would you evaluate the mirror in the field? What data would you collect? Over what period?

Re / Inducing Sickness

Human beings tend to believe what their eyes tell them over what other senses are telling, so by giving stimulus to the eyes we can 'fool' the brain. Obvious cases are TV and theatrical films where we believe we see motion from a sequence of still frames. Video games add interaction to this mix which can make certain cues very powerful. e.g. video games can give a player a strong sense of vertigo while standing on the edges of cliffs or walking across thin bridges over chasms. Done right, these inputs can enhance the realism of the experience, but done wrong they can affect the user more seriously.

Carpal Tunnel Syndrome / Repetitive Stress Injuries

2 things are needed: a functioning vestibular system (canals in the inner ear) and a sense of motion

Symptoms: Nausea, eyestrain, blurred vision, difficulty concentrating, headache, drowsiness, fatigue

These symptoms can persist after the experience is finished.

Causes: still unknown but most common hypothesis is a mismatch between visual motion (what your eyes tell you) and the vestibular system (what your ears tell you)

Why would this cause us to become sick? Possibly an inherited trait - a mismatch between the eyes and ears might be caused by ingesting a poisonous substance so vomiting would be helpful in that case.

• sense of motion is required
• bright images are more likely to cause it than dark ones
• wide field of view is more likely to cause it than narrow field of view
• systems without a 'rest frame' are more likely to cause it
• low resolution, low frame rate, and high latency are also likely causes

fighter pilots have 20 to 40 percent sickness rates in flight simulators - but experienced pilots get sick more often than novice pilots.

In a rotating field when walking forward, people tilt their heads and feel like they are rotating in the opposite direction.

If a person is walking on a treadmill holding onto a stationary bar and you change the rate the the visuals are passing by, it will feel to the person like the bar is pushing or pulling on their hands.

Its easy (and fun) to induce vertigo.

vection - A person viewing a display with moving objects often feels that he/she is moving in the opposing direction.

Sickness is reduced when you have a stable reference frame (i.e. being able to see the border of the screen.) This is why in most current HDM VR games that involve flying you get a stable cockpit around you. Most current Head Mounted Display games don't let you use your controller to 'run' through a scene and only let you teleport to reduce the chances of getting sick. When moving in the HMD version of Google Earth they reduce the field of view to remove motion in the periphery of your vision for the same reason.

Nintendo released their first VR game console, the Virtual Boy, in 1995. They decided not to make it a Head Mounted Display for safety reasons - not wanting people to run into things or fall down the stairs while wearing it. The console also forced the player to stop playing every 15 minutes or so to rest their eyes.

The Pokemon Incident 

December 16 1997

685 schoolchildren taken to hospitals- feeling sick while watching Pokemon

12 Hz red - blue flicker scene lasting about 5s roughly 20 minutes into the program

Show aired in several major cities (Tokyo, Osaka, etc) and then excepts were shown on the nightly news after reports came in - causing more cases. Broadcast of the show was cancelled in 30 other cities.

Pokemon incident was the first occurrence on a mass scale

New type of trigger, not just rapid light/dark - this is now known as "chromatic sensitive epilepsy."

The episode was pulled, but of course many people had their VCRs (Video Cassette Recorders) running so copies exist and these days you can find it on YouTube in various places by searching pokemon epilepsy - the videos keep getting removed and then added again with different links.

https://www.youtube.com/watch?v=Wu-KAY3lJ6A

Pokemon on the brain

Back in 2008 there was a lot of coverage on the internets about some rather nasty things posted to the Epilepsy Foundation's message boards - very simple animated GIFs could cause people to have seizures.

20 years after the Pokemon Incident there have been a series of Pokemon Go related incidents including traffic accidents (driving while playing and walking while playing), walking / falling accidents, robberies / assaults, etc. as people are concentrating on their phones and not on their surroundings. Similar things happen with texting. The game or application itself is not directly to blame for the injury, but is involved, and as we move towards augmented reality displays, and 'always on' and 'always visible' information flows we have to help people balance their lives in the real world and the virtual world that we are all increasing simultaneously inhabiting.