Evaluation Techniques and Re/Inducing Sickness

(material from: Information Anxiety by Saul Wurman, The Psychology of Human-Computer Interaction by Stuart Card and friends, Designing the User Interface 3rd Ed. by Ben Schneiderman, Human-Computer Interaction 2nd by Dix etc)

3 goals

Jonas Salk spent 98% of his time documenting things that didn't work before he found the thing that did

Kenneth Boulding "The moral of evolution is that nothing fails like success because successful adaption leads to the loss of adaptability ... This is why a purely technical evaluation can be disastrous. It trains people only in thinking of things that have been thought of and this will eventually lead to disaster"

How evaluation is done depends on many factors

Different types of Evaluation

Evaluating the Design

   cognitive walkthrough

    heuristic evaluation

    review-based evaluation
            look through existing literature for previous related experiments

    model-based evaluation
            GOMS, keystroke-level model, etc

Evaluating the Implementation


    Expert reviews

Usability Testing


    Acceptance Tests
         establish specific testable criteria for the application:
            time to learn, speed of usage, rate of errors

Controlled Experiments

We can also get a lot of information using eye-tracking systems telling us where people are actually looking in addition to mouse clicks and touches telling us where they are interacting.



and eye tracking is getting cheaper and more available with gadgets like the Tobii

examples: https://www.youtube.com/watch?v=Rjha0nKbwnk and https://www.youtube.com/watch?v=PId82ll7KrM

and also more and more data tracking users positions, gestures, etc.

Video and Audio recording, especially when a subject is speaking aloud, can be incredibly useful for finding interesting subtle things, but it can take a very long time to go through. Early on in evaluation you are often looking for interesting events or sequences of events. In a more refined study you have a set of predefined codes for particular events and go through the video and code the time in the video you saw each particular event that you are interested in.

Its very important to treat your users / test subjects well

IRB (Institutional Review Board) issues

and especially issues of informed consent

here is a consent form and media consent form that we used last year

In-class group assignment time

Send andy an email or a link to a web page discussing how you would study the usability of your mirror.

Part 2 - What are the possible dangers in using your mirror and how are you minimizing those risks through your design and implementation?

Re / Inducing Sickness

Human beings tend to believe what their eyes tell them over what other senses are telling, so by giving stimulus to the eyes we can 'fool' the brain. Obvious cases are TV and theatrical films where we believe we see motion from a sequence of still frames. Video games add interaction to this mix which can make certain cues very powerful. e.g. video games can give a player a strong sense of vertigo while standing on the edges of cliffs or walking across thin bridges over chasms. Done right, these inputs can enhance the realism of the experience, but done wrong they can affect the user more seriously.

Carpal Tunnel Syndrome / Repetitive Stress Injuries


'button mashing' injuries up - 'falling out of tree' and 'riding a bike' injuries down among children in the last 15 years.

a more recent one is repetitive stress injuries due to people playing with their wii when it first came out. For example if you play real tennis and swing the racket you (usually) hit the ball and transfer some energy to the ball. Playing wii tennis you need to slow your arm down without that transfer of energy when you 'hit' the virtual ball.

a recent one is hearing loss from portable music players

Why do you think this wasn't this as much of a problem with portable CD players in the 90s or cassette players in the 80s?

Simulator Sickness

2 things are needed: a functioning vestibular system (canals in the inner ear) and a sense of motion

Symptoms: Nausea, eyestrain, blurred vision, difficulty concentrating, headache, drowsiness, fatigue

These symptoms can persist after the experience is finished.

Causes: still unknown but most common hypothesis is a mismatch between visual motion (what your eyes tell you) and the vestibular system (what your ears tell you)

Why would this cause us to become sick? Possibly an inherited trait - a mismatch between the eyes and ears might be caused by ingesting a poisonous substance so vomiting would be helpful in that case.

fighter pilots have 20 to 40 percent sickness rates in flight simulators - but experienced pilots get sick more often than novice pilots.

In a rotating field when walking forward, people tilt their heads and feel like they are rotating in the opposite direction.

If a person is walking on a treadmill holding onto a stationary bar and you change the rate the the visuals are passing by, it will feel to the person like the bar is pushing or pulling on their hands.

Its easy (and fun) to induce vertigo.

vection - A person viewing a display with moving objects often feels that he/she is moving in the opposing direction.

Sickness is reduced when you have a stable reference frame (i.e. being able to see the border of the screen.) This is why in most current HDM VR games that involve flying you get a stable cockpit around you. Most current Head Mounted Display games don't let you use your controller to 'run' through a scene and only let you teleport to reduce the chances of getting sick. When moving in the HMD version of Google Earth they reduce the field of view to remove motion in the periphery of your vision for the same reason.

Nintendo released their first VR game console, the Virtual Boy, in 1995. They decided not to make it a Head Mounted Display for safety reasons - not wanting people to run into things or fall down the stairs while wearing it. The console also forced the player to stop playing every 15 minutes or so to rest their eyes.

The Pokemon Incident 

December 16 1997

685 schoolchildren taken to hospitals- feeling sick while watching Pokemon

12 Hz red - blue flicker scene lasting about 5s roughly 20 minutes into the program

Show aired in several major cities (Tokyo, Osaka, etc) and then excepts were shown on the nightly news after reports came in - causing more cases. Broadcast of the show was cancelled in 30 other cities.

Pokemon incident was the first occurrence on a mass scale

New type of trigger, not just rapid light/dark - this is now known as "chromatic sensitive epilepsy."

The episode was pulled, but of course many people had their VCRs (Video Cassette Recorders) running so copies exist and these days you can find it on YouTube in various places by searching pokemon epilepsy - the videos keep getting removed and then added again with different links.


Pokemon on the brain

Back in 2008 there was a lot of coverage on the internets about some rather nasty things posted to the Epilepsy Foundation's message boards - very simple animated GIFs could cause people to have seizures.

20 years after the Pokemon Incident there have been a series of Pokemon Go related incidents including traffic accidents (driving while playing and walking while playing), walking / falling accidents, robberies / assaults, etc. as people are concentrating on their phones and not on their surroundings. Similar things happen with texting. The game or application itself is not directly to blame for the injury, but is involved, and as we move towards augmented reality displays, and 'always on' and 'always visible' information flows we have to help people balance their lives in the real world and the virtual world that we are all increasing simultaneously inhabiting.

last updated 4/14/2017