We envision situation-rooms and research laboratories in which all the walls are made from seamless ultra-high-resolution displays fed by data streamed over ultra-high-speed networks from distantly located visualization, storage servers, and high-definition video cameras.

SAGE allows local and distributed groups of researchers to work together on large amounts of distributed heterogeneous datasets. From our prior work on the Continuum, we learned that it is crucial for collaborators to have both local control (e.g. on a tablet or laptop) and the casual ability to share their work and see what others are working on (e.g. on a large tile display). We are taking the next steps toward this vision by building LambdaVision - an 11x5 tiled display with a total resolution of 100 megapixels, and developing SAGE as shown in the following figure. High-resolution displays like LambdaVision are necessary to support geoscientists working with aerial and satellite imagery (365Kx365K pixels maps) and neurobiologists imaging the brain with montages consisting of thousands of pictures from a high-resolution microscope (4Kx4K pixels sensor).

The Free Space Manager (i.e. window manager) in SAGE provides an intuitive interface for moving and resizing graphics on the tiled display. When a graphics window is moved from one portion of the screen to another, the Free Space Manager informs the remote rendering clusters of the new destination for the streamed pixels, giving users the illusion that they are working on one continuous computer screen, even though each of their systems may be several thousand miles apart. Last but not least, Free Space Manager is akin to a traditional desktop manager in a windowing system, except that it can scale from a single tablet PC screen to a desktop spanning over 100 million pixel displays.

FRAMEWORK

SAGE consists of Free Space Manager, SAGE Application Interface Library (SAIL), SAGE Receiver, and User Interface (UI client) as shown in the figure below. The Free Space Manager gets user commands from UI clients and controls pixel streams between SAIL and the SAGE Receivers. SAIL captures output pixels from applications, and streams them to appropriate SAGE Receivers. A SAGE Receiver can get multiple pixel streams from different applications, and displays streamed pixels on multiple tiles. A UI Client sends user commands to control the Free Space Manager, and receives messages that inform users of the current status of SAGE.

Free Space Manager is the central control unit of SAGE. It communicates with UI clients by receiving user commands and sending SAGE status messages. When the Free Space Manager receives user commands, it sends appropriate control messages to SAIL and the SAGE Receivers to perform various SAGE functions such as application execution, window movement and resize, and window z-order change (for overlapping windows). When functions are completed, SAIL and SAGE Receivers inform the Free Space Manager to update the status of SAGE to the UI clients, such as new window position or scale. Free Space Manger is also an information server for SAGE, and contains several possible execution configurations of SAGE applications, run-time information of each application instances, and tile display configuration. Free Space Manager provides SAGE components with information whenever needed in order to perform SAGE functions.

SAIL is the library that SAGE applications use to communicate with Free Space Manager, and to stream pixels to SAGE Receivers. When a SAGE application is executed, it creates and initializes SAIL objects – one SAIL object per each node, if the application is running on multiple rendering nodes. To initialize SAIL, each application node has to give it the pixel format – 24bit RGB, 32bit RGBA, 16bit RGB, etc – and information about the image rendered by the node –resolution of the image and the image's coordinates in the whole application image. Then, SAIL connects to the Free Space Manager, and the Free Space Manager informs SAIL how it should split the application images and where to stream the split images. Whenever a SAGE application gives SAIL a new frame by calling a "sageSwapBuffer" function, SAIL splits and sends the new images to the proper SAGE Receivers.

SAGE Receivers get pixel streams from SAIL and control the tiled display residing on the display nodes. They are remotely launched by the Free Space Manager and receive all the information about the tiled display from it. Each display node has only one SAGE Receiver, whereas more than one SAIL object can exist on a rendering node, since they are created per each application instance. Hence, a SAGE receiver has to be able to receive multiple pixel streams from different application instances or different rendering nodes. Additionally, it has a circular buffer for each stream so that the stream receiving threads fill the buffer, the displaying threads read the buffer, and display the pixels on the tiled displays. The current version of SAGE does not require applications to change the rendering resolution when users resize the application windows. Instead, SAGE Receivers scale the applications' images to fit the new window size.

UI Clients send user commands to Free Space Manager and show the status of SAGE to users. UI clients can be a Graphical User Interface, text-based console, or tracked devices. We have already implemented a text-based console and the first version of the SAGE GUI that can execute, move, and resize SAGE applications by sending commands to the Free Space Manager. UI clients can reside on any machine that can be connected to the Free Space Manager by networks, and multiple UI clients can connect to a Free Space Manager and interact simultaneously. This allows multiple users to connect to SAGE (a Free Space Manager) from their laptops, tablets, or desktops and to control SAGE, while interacting with each other.

^

USER INTERFACE

The SAGE UI is designed to be a cross-platform graphical user interface window manager for SAGE. Instead of relying on fsConsole for manipulating windows on the SAGE display, a graphical representation of the tiled display configuration and the location of the window can be directly manipulated by the user from his/her machine. One of the nice features of SAGE is that communication between any SAGE client and fsManager is done using text messages. By using a text-based communication method, different types of SAGE clients can interact with fsManager, thus allowing great flexibility in the programming language used to create the SAGE client. For instance, it may eventually be possible to translate hand gestures from a user, as captured by a camera, in order to move windows directly, while other users control the SAGE display using a more traditional user interface. Furthermore, since SAGE was designed with collaboration in mind, multiple SAGE UIs can be used to control one SAGE environment at the same time. Each SAGE UI will show the same state, being the current state of the tiled display.

SAGE interface showing Application windows positioned in a 2x2 tiled display, underneath are the applications menu and performance data.

^

LAMBDATABLE

The Lambda Table is a user interface system developed to facilitate multi-user interaction over a common high-resolution horizontal tiled display surface. To give a more direct sense of interaction with the table display, we have developed a system to track physical user interface icons, or "phicons," placed directly on the table by using an overhead camera. This system, when integrated with SAGE, will provide another means for researchers and collaborators to view and manipulate large scale distributed datasets. The Lambda Table also opens up new possibilities for group based data-centric interaction, and provides a native environment for traditionally horizontal media such as maps or geographic data.

The current implementation is a flat arrangement of 15 tiled LCD monitors with a plexiglass sheet on top. A camera mounted overhead tracks the motion of the phicons which are uniquely identified by a pattern of infrared LEDs. The position and orientation of the objects are calculated and sent over the network to an application running on the table, such as SAGE. Further research is underway to explore new human interface devices that may present wholly different methods for interacting with data. Devices such as physical markers, zoom widgets, and magic windows will offer researchers more natural tools for searching, labeling and sharing large sets of information over a common horizontal surface.

^