Implementation of a virtual therapist in a home-based rehabilitation system for stroke survivors

Fulbright Scholar Project September 2011-January 2012
Institute Image-Arts et Métiers ParisTech, Chalon-sur-Saône, France

Deliverables

A prototype developed using 3DVIAVirtools software (Dessault Systems, France)
video documentation

A module developed using the Unity3D software (Unity Technologies)
video documentation

The goal for my research in France was to develop a networked, multiuser (therapist and patient) virtual reality home rehabilitation system for stroke survivors. The therapist and the patient could be physically located within their own homes while interacting with each other in the online virtual scene. For the patient this can be much easier, more comfortable, fun, and less intimidating to interact with the 3D character March Hare– therapist’s avatar online than being in a clinical setting with a real person guiding every movement.

This system is designed to engage stroke patients and encourage them to manipulate their impaired arm and hand to foster repetitive practice of hand movement. The central narrative draws upon the classic story of Alice in Wonderland and describes a tea party in the cottage of the March Hare. In this story, a series of tabletop activities of daily living are introduced but with a “mad tea party” twist that creates the opportunity for inventive repetitions of rehabilitation tasks and special effects. The focus is placed on the setting as a magical place where anything can happen. For example, cookies morph into crabs and scurry away from the participants; bluebirds fly from the china pattern and hover above the table, etc. Therefore the users must employ reach-to-grasp movements to grab the crabs, catch the birds, or rotate their arm to drink the wine. Consequently, movement practice is encouraged as the motivation for participation comes from the user’s engagement with the narrative and the environment.

The project has two deliverables; the module being developed using a free version of the Unity3D software (Unity Technologies) and a module being developed using 3DVIAVirtools software (Dessault Systems, France). Both modules have the same goal – the development of the online multiuser virtual environment for remote hand rehabilitation therapy. However, the final output differs as Unity 3D currently does not support a stereoscopic output while the Virtools module does.

The project has a functioning multiuser scene which can be shared virtually online and in which therapist and patient see each other as 3D characters / avatars. I have implemented two avatars; the therapist avatar (March Hare character) and the patient avatar (Casual young male character) in the multiuser scene. The magnetic arm trackers (Flock of Birds, Ascension Technologies, Burlington, VT) measure the position and orientation of avatars hands to allow therapist and patient interact with the scene. Their virtual arms and hands move in accordance with the user’s actual movement. The tea party scene is updated according to the orientation and position of the head, as measured with a secondary magnetic head trackers (see Fig. 2).

Figure 2. Schematics of the multiuser system architecture. Note: the HMD and the PneuGlove are not included in the prototype documentation.

The therapist starts the private virtual therapy session online on his computer. The therapist is the session master and can lock the session or end it at any time. The patient defines his/her user name and connects to open sessions online using the session ID and his/her login/password. The therapist knows when the patient joins the session as the patient’s avatar immediately appears at the table scene. The universal time synchronizes the actions between the patient and therapist interfaces. The therapist can monitor the patient’s presence and movements in the session. If the patient leaves the session, the patient’s avatar will disappear from the virtual scene.

As the therapist shows the patient how to do the rehabilitation exercise, his hand gestures, motion path and orientation are replicated in real time in the virtual environment, enabling the patient to see and repeat them. The March Hare avatar directs the client visually through the therapy tasks. The Inversed Kinematics (IK) feature was implemented to support naturalistic gestures to accompany the rehabilitation task instructions. The IK animations are based on the enhancement of the avatar skeletons which I done using Maya (Autodesk) 3D software.

The patient and the therapist can type in and read text messages in the chat window and communicate in real time. As each user presses the “enter” key, the text message appears on top of his avatar instantaneously. In addition, the therapist and the patient can communicate via an audio software application that allows users to make voice over the Internet (Skype, Microsoft). Previous studies showed that audio communication positively affects the user's experience of social presence and collaboration in a virtual environment.

Currently four different training exercises having been implemented, including capture of the birds, taming bouncy sugar cubes, drinking wine and finger-painting. These tasks focus on different types of hand movements, such as whole-hand grasp, flat palm, and arm stretch. I have implemented special tools which therapist can use to explain the rehabilitation exercises. Visible tracing tracks the therapist hand movement with a virtual line visible to the patient and the therapist in the finger painting exercise. Both therapist and patient can see the actual hand path drawn and compare it with the patient’s path, which can also be drawn in the scene. The therapist is also able to invoke a “mirror mode.” In this mode, the therapist performs movements, which the patient attempts to mimic.  The March Hare, the therapist’s avatar, shows the desired movement and then can watch the patient to assess performance.