| Josephine Home | Thing home |
In the following document I will describe the content of this project, discuss some of the technical details and end with an account of the MFA show.
The user starts out on a plain. Clouds pass overhead singing. A voice-over suggests that the user go towards a small shed in the distance. The user navigates through the world with a 3D joystick. As the story begins the ideal user is enjoying the sense of agency that navigation brings, exploring curiously, but open to the suggestion to go to the shed!
Inside the shed is a box. A squeaky voice comes out of it saying, "Please let us out, we're poor little kittens. We're trapped!" At first the user is powerless, then a key appears. If the user doesn't do anything it prompts the user to click on it. When the user clicks, the key moves to the box, turns in the lock, then...
CATTASWOOSH!
The shed explodes as the box flies open and dark rock-like shapes pour out of it and scatter across the plain. Before the user can draw breath, the rest of the box collapses and the Thing pops out joyfully shouting, "I'm free! You freed me! I love you!" The Thing is a creature composed of multi-colored, translucent pyramids. It is immediately obsessed with consummating its feelings for the user by teaching the user a dance.
Act 2
The Thing encourages the user to move her physical arms and body, and dance. Information from the trackers the user wears, tell the Thing if the user is really trying. Information from the joystick reveals whether the user is paying attention and learning or sneaking away. The Thing's intelligence then picks a response to the user; praising or complaining, teaching a new dance step, going over a step copied badly. In this act the interaction is designed to be so natural as to be invisible. The crucial element is that the Thing's reactions fit the user's actions.
The visual and sound design of the piece convey a cartoony, fairy-tale innocence. The visuals are primitive, the sound design hearkens back to radio stories, with the narrator creating the special effects as well as all the voices. This innocent facade is designed to win the user's trust. At first the ideal user thinks the Thing is cute and funny and is willing to try and dance. The fact that it pretended to be a sweet little kitten to get out of the box is the first hint, however, that it is a duplicitous Thing.
And so the story line moves along. As the dance lesson continues the Thing reveals more and more of its character. It will flatter, coax, whine, beg, or threaten to get its way. And it gets pickier and pickier about the dancing. The ideal user feels more and more cramped and overwhelmed by its demands. But if she tries to run away, the Thing follows crying or shouting, insisting on its love, insisting on the dance.
The story moves on to the next stage depending on the user's attitude or on a time interval. The Thing becomes enraged either at the user's refusal to dance or her inability to dance well enough and storms off. As soon as the Thing has gone, the rocks that burst out of the box onto the plain start moving. They hustle around the user, getting closer and closer, herding her. When one get close enough it will rear up and try to grab her. Eventually one of them succeeds. The user finds that her navigation is disabled and she is trapped beneath a slurping rock. The Thing returns to tell the user, that the rock is dripping acid on her, in preparation for eating her. However, if the user is nice to Thing - i.e. dances with it - it will get her out.
This creates a symmetry in the story-line. First the user released the Thing, now the Thing releases the user. The second act as a whole is designed to make the user wary of the Thing. It's behavior has undercut her initial assumptions. It can be nice but it is clearly manipulative and dominating.
In that last section of the second act the Thing shows its amiable side again by suggesting that it will copy the user's movements. As the user moves her tracked arms and head the Thing mimics her crooning all the while about their synergy. Suddenly lightning and thunder fill the plain, and a disembodied voice shouts, "What's going on here?" A bolt of lightning strikes a the user and the ground beneath her feet opens.
Act 3
The Thing and the user fall helplessly into a new environment which is red and claustrophobic dotted with strange menhirs. The user is again trapped, this time in a claw-like cage. Four beings similar to the Thing but differently colored advance. They greet the Thing, calling it "cousin." The Thing cowers before them and whispers to the user that it is heresy for the two if them to dance together.
The cousins turn on the Thing and accuse it of disgusting behavior that defiles all Thing-kind. It has entered a relationship with a meat object - the user! The cousins beat the Thing severely, then throw it up into the cage and exit mouthing dark threats. The Thing is crying and hurt. It apologizes to the user but then reveals it has a contingency plan. It has concealed a weapon.
The Thing pops a gun out of an orifice and instructs the user to pick it up and blast them out of the cage. As they escape the four cousins reappear, and the Thing encourages the user to chase and shoot them also. Then comes a moment when all the cousins are dead or escaped and the Thing stands in front of the user and the gun. Suddenly it fears that the user will shoot it too!
Everything depends on the user's feelings for the Thing. And by this point the ideal user is ambivalent. In act three she and the Thing have been thrust into danger and onto the same side. They have mutually helped each other. But she can also remember the Thing's bullying ways.
The moment of decision is both the ending and beginning. The piece is designed to be circular. If the user doesn't shoot - she is returned to the welcoming Thing, who now assumes that she is ready and willing to dance the dance forever. If she shoots, she is returned to the beginning, the voice over tells her she can go to the shed and release the Thing or remain on the plain alone, forever.
The Thing Growing is an exploration of the mis-use of power and love
in relationships, and of our
tendency to repeat dysfunctional relationships trying to "master"
them.
The user has between two and four trackers attached to head, arms and body, and holds a 3D mouse with three buttons and a joystick for interaction and navigation. Information from the trackers, joystick and buttons is used by the application to figure out where the user is and what they are doing.
This application is programmed using the CAVE library, Performer and XP. XP is a VR authoring system based on Performer designed by Dave Pape. Other VR projects programmed in XP include, the Multi-Mega Book in the CAVE, and Mitologies.
XP
Among other things, the basic XP system takes care of loading and transforming models; collision detection; navigation; triggering events; integrating sound and switching individual objects or whole scenes on and off. In addition messages can be sent from any object in the scene graph to any other object.
To create the virtual scenes XP reads a hierarchical text file. Objects in the text file can be models or behaviors. Messages and event triggers, (based on time, distance or other parameters) can all be entered into the text file. By editing this text file non-programmers can swiftly construct complex, interactive environments and alter them easily.
The XP system is also easily extendible for programmers. Classes with specific behaviors can be added to the central core. Each new class inherits from the XP node class, so that its attributes are parsed by the XP system. The messages it understand, and events that it triggers are added into the system. The classes do not have to know about each other to effect each other. In the scene file we can link any object's event trigger to any message for any object.
Examples of Use and Extension of XP for The Thing Growing
The XP message system was invaluable for constructing the narrative backbone of the Thing Growing. In the first scene, a distance sensor triggers a message to open a door; a push of the button sends a message to activate a key; delayed messages blow up a box and shed. At a later point in the story the Thing becomes so angry with the user that it hides under one of the rocks that dot the plain the initial action takes place on. At this point the Thing object sends out a message to all the other rocks that it's time to catch the user. This message triggers a set of stalking and herding behaviors in all the rock objects.
Intelligence has to be programmed into the rocks - they have to know where the user is, they have to avoid each other, and they have to sneak up on the user and try to trap her. Dave extended the basic XP transform class and made a rock-object class which kept a list of all other rock-objects and programmed them with a set of rules on how to move until one grabs the user. When that happens, all the other rocks scatter and a message is sent to the navigator to disable navigation. The user is trapped with a rock slobbering on her.
Much of the extension of XP for this application lay in creating classes with autonomous behavior for moving objects in the VR environment. Most specifically for the creation of the Virtual Character, the Thing.
The Thing: Moving the Thing's Body
The Thing is a collection of detached translucent triangles, one for the head, two for arms, one for a body and four or five for a tail. It is animated using motion tracking. The life-like movement that results causes the user to sketch in the lines of a dragon-like creature.
The Thing is a speaking creature. Based on the initial story board, I recorded a library of short phrases for its voice. Sometimes the things it says are scripted and do not vary - for example when it is freed from the box it is trapped in. But mostly it has to speak in response to the user - for example when it is teaching the user to dance. For this section I recorded different versions of words of praise, encouragement, criticism, explanation. I also recorded these different types of utterance in different moods - happy, manic, sad and angry. Each phrase lasts a few seconds. The animation was done in time to these phrases.
Dave built an XP recording tool for use in the CAVE. To use it I make a text file loaded with sound files and the Thing's body parts. I enter the CAVE and attach trackers to my head, arms and body. In front of me I see the model of the Thing, around me boxes represent the sound bites. My motions are directly relayed to the model so that it moves in the same way I do. I click on a sound bite box. That particular sound plays back while I move and watch the Thing moving. I try to create a motion that captures the sense and mood of the words. Buttons on the wand allow me to record my movement, then play it back. During this play back I can both hear the sound bite and walk around the Thing watching it make the movement I just recorded. If necessary I can re-record the movement.
Since we only have four trackers to work with - I record the tail in a separate step. The tail is a spring - Dave wrote an XP spring class - attached to the body at one end. I manipulate a tracker that feeds information to the other end of the tail. I play back the body motion and then run around with the tail-end, trying to make an appropriate tail motion. The central pieces of the tail move according to the spring calculations between the two fixed points.
The position and orientation information obtained in this process is stored in text files. When the application is running, the computer interpolates between the end of one movement and the beginning of the next, so that the motion is smooth - Dave wrote all the code for making the body parts move correctly.
As well as this animated movement for its body parts. The Thing needs to move its body as a whole with reference to the environment and the user. At times it needs to stays close, swoop in on, or run after the user. At other times it needs to move to a particular spot in the environment. I wrote another class extending the basic XP transform class to execute this global body behavior.
The Thing: Intelligence
Essentially programming for the Thing has two major categories - body and brain. My focus was on writing the code for the brain.
The brain has access to stores of all the actions that the Thing knows. Each action consists of three parts; a text file with the motion capture information; a phrase that goes with this motion; and a message to tell the global body how to move the body as a whole. The brain's job is to select an appropriate action according to the point in the narrative, the user's actions, and the Thing's own emotional state, and to pass the information onto the Thing's body parts, to its voice and to its global body so that they will all execute this action.
The narrative constrains the number of actions and action stores we need to create for the Thing. For example when the Thing is attempting to teach the user to dance, it has a basic routine to follow:
This routine is interrupted if the user tries to run away. Then
a different routine is triggered to make the Thing run after the user and
plead with her or scold her to continue the dance.
Essentially each type of response - "demonstrate a step", "dance with user", "run after user", "praise user" - has an action store filled with possible actions. When one action is finished the intelligence performs some checks and decides on the next action. Then the action is pulled out from the appropriate bin and sent to the global body, body parts and voice for execution. The action can be pulled out sequentially, at random or by mood. (Remember the Thing can respond in one of four moods, angry, sad, happy, manic.) The checks include seeing whether the user is navigating, whether any buttons are being pressed, and how much the user is moving her limbs.
By the summer of 1999 I had a first version of the complete story working. Feedback from my committee and observation of users indicated several major problems that needed to be addressed for the project to successfully communicate its story.
First some users had difficulty discerning the Thing's body. It is a collection of pyramids "tied" together my the motion captured movement. I expected people to move their arms in the same way that the Thing moved it's arms. Some users, however, simply wriggled about like hypnotized snakes - I realized that they were trying to copy the Thing's tail movements. Adding a verbal injunction to "move your arms and body like mine - don't worry about the tail," did not fully fix the problem. Dan Sandin suggested outlining the Thing's body and de-emphasizing the tail. This new look helped people to correctly identify the body and move their own bodies' accordingly.
A second and related problem was to clearly indicate to people that they had to move their bodies during the dancing sequence, and that the Thing "knew" what they were doing. Some user's, especially expert users, thought that they should use the joystick and buttons to dance - this resulted in the Thing rushing after them and chiding them for moving away. They would then try to use the joystick to drive close to the Thing, which it would continue to interpret as "driving away" behavior. Confusion would ensue with the user muttering, 'I'm just trying to get close." Other users would only dance with one arm - although in the final version both arms are being tracked.
To solve these problems I inserted some new behavior at the very beginning of the dance sequence. The Thing announces that it will show the user how to dance, then says that they must "loosen up a little first." It instructs the user to wave her arms above her head. The program performs checks to see :
Three other problems were more to do with content than technical issues. In the first draft the Thing was simply too unpleasant and people nearly always shot it as soon as they were given the chance. Ideally I wanted the user to be more ambivalent. There was also a timing problem. In the second act the Thing danced with the user, then stormed off in a huff. The rocks on the plain then chased and trapped the user. The user had barely been released from that trap when the third act was triggered and the user was again trapped while the four cousins berated the Thing. This period of inactivity (in-interactivity) for the user, was too long. Third, the transition between the second and third acts was too abrupt - people were confused about what had happened and did not understand who the cousins were. Some users believed that they had come to save them from the Thing, rather than feeling that they were now on the same side as the Thing and in deep trouble.
The solutions for these problems were interdependent, but I will describe them as they happen chronologically in the story. First I added a new behavior in which the Thing now copies the user's movements and inserted this after the user is released from the rock. The Thing says, "Let's do something different - you dance and I'll copy." The user discovers that as she moves her arms and head the Thing moves with her. Basically we took the tracking data from the user and applied it to the Thing's body with a slight delay. It is strangely flattering - because the Thing becomes a mirror of our own idiosyncratic movement. The Thing becomes very seductive while this is going on. And the user feels that she is now in control. So this small new scene, lasting about 40 seconds serves two purposes. It makes the user "like" the Thing more - we like the thing that does what we want - and it breaks up the two periods where the user is trapped and unable to interact.
Second I added a more elaborate transition scene between the second and third acts. The user is now happily dancing with the Thing copying her. Suddenly lightning crashes across the sky, the sky darkens and a god-like voice out of nowhere booms, "What's going on here?" Another higher voice cries, "We must stop this evil!" A lightning bolt flashes towards the user, cracks appear in the ground and the user and the Thing fall through a red trench into a world below. This is the territory of the four cousins.
Now as the four cousins approach, the Thing whispers asides that reveal that the dancing behavior that they were indulging in is heresy if it takes place between a meat-object, the user, and a Thing. As before the four cousins berate the Thing, but now they are more vicious and pull off its arm. This is to encourage the user to feel more protective of the Thing.
During the sequence where the user shoots at the cousins, all the Thing's phrases were revised. First several off them give more information about the back story- the taboo relationship which user and Thing have been caught in. Second the Thing's mood and behavior alter depending on the user. If the user can't or won't shoot the cousins, the Thing begs her to protect it, and finally becomes furious and abusive. On the other hand if the user does shoot the cousins, and the faster she shoots, the Thing begins to have second thoughts and begs the user to spare her family members. All this is designed to increase the user's ambivalence.
Feedback from users and my committee indicated that these fixes, which comprised a second draft of the project, conveyed the story much more completely. The entire experience in this final version lasted between 10-15 minutes depending on the user.
Show time
On the evening of March 9th and during the day on March 10th, 2000, I had my thesis show. The Thing Growing showed every 20 minutes in two CAVEs and on two ImmersaDesks simultaneously. Having four VR devices maximized the number of people who could directly interact with the application. This was important to me because the impact of the piece is very different when you are the one the Thing addresses and reacts to.
With the four VR devices running a maximum of 20 people could see the project at any one time, 8 in each CAVE, 3 at one of the Idesks and 1 at the other. At the two CAVEs and at one of the Idesks the tracked user had an audience. This user's emotional and psychological responses are not only affected by the Thing but by being watched by others.
At the second Idesk one user, wearing headphones, went through the experience alone. This user was very much enclosed in the world with the Thing, and without onlookers. Ideally I would like the piece to be this kind of one on one experience all the time. The second day was less crowded and we showed in one CAVE only. Which meant that a couple of people could have the intimate one on one experience in the larger immersive environment.
In the two days about 200 people saw the piece. This was only possible because about twenty people helped organize and man the show.