Understanding CAVERNsoft

This page is split into two main sections: an introduction to CAVERNsoft and the quick pictorial outline on how one typically codes with CAVERNsoft.

Introduction to CAVERNsoft

CAVERNsoft is most succinctly described as a C++ hybrid-networking/database library optimized for the rapid construction of collaborative Virtual Reality applications. Using CAVERNsoft VR developers can quickly share information between their applications with very little coding. CAVERNsoft takes over the responsibility of making sure data is distributed efficiently.

The IRB

The Information Request Broker (IRB) is the nucleus of all CAVERN-based client and server applications. An IRB is an autonomous repository of persistent data driven by a database, and accessible by a variety of networking interfaces. The goal is to develop a hybrid system that combines a distributed shared memory model with distributed database technology and realtime networking technology under a unified interface to allow the construction of arbitrary collaborative VR (CVR) topologies.

A client application is built by using an IRB interface (IRBi) which, on invocation, will spawn the client's ``personal'' IRB. This IRB is used to cache data retrieved from other IRBs during the operation of the client. An application specific server is similarly built using the IRBi. Hence there is actually little differentiation between a client and a server. Using the IRBi a client can arbitrarily form a connection with any other client or server to access its resources. The IRBi will communicate the request to the client's personal IRB which will then communicate with the remote client's or server's IRB. It is the IRBs' responsibility to negotiate the networking and database services requested by the client/server applications. This form of flexibility will allow arbitrary CVR topologies to be constructed.

The IRB Interface

The IRB interface (IRBi) is the client and server's interface to the IRB. The IRBi provides a networking interface, a database interface and a high-level template interface. The IRBi is tightly coupled with the IRB as they are merely threads that share the same address space. This reduces the need for creating artificial message passing schemes to invoke functionality between the client's application and the IRB.

A client's handle to their personal IRB is used to activate dynamic connections with remote IRBs. A client wishing to share information between its personal IRB and a remote IRB begins by first creating a communication channel and declaring its communication properties. Then any number of local and remote keys may be linked over the channel. A key is a handle to a storage location in an IRB's database. The database is used to cache data received from remote keys. Keys are uniquely identified across all IRBs and can be hierarchically organized much like a UNIX directory structure. Each local key may be linked to only one remote key. This is the link explicitly invoked by the user to share information with a remote IRB. However each local key can accept multiple linkages from other remote subscribing keys. The user is generally made unaware of these additional linkages as the personal IRB transparently manages data sharing with the remote subscribers. Any modifications that are made to one key will automatically be propagated to all the other linked keys.

Channel Properties

Channel properties allow clients to specify the networking service desired for data delivery. Clients may specify reliable TCP, or unreliable UDP and multicast. Large packets delivered over unreliable channels will automatically be fragmented at the source and reconstructed at the destination. If any fragment is lost while in transit the entire packet is rejected.

In addition to connection reliability clients may specify Quality of Service (QoS) requirements. Hence they are able to declare the desired bandwidth, latency, and jitter of the data stream. The personal IRB will attempt to obtain the desired level of QoS from the remote IRB, but if it fails, the client may at any time negotiate for a lower QoS.

Link Properties

Link properties allow clients to specify the actions taken when local and remote keys are linked. This includes being able to choose between active and passive updates and being able to select the initial and subsequent synchronization behavior.

In most CVR applications, world state information consisting of a few tens of bytes are actively distributed. That is, the moment a new value is generated it is automatically propagated to all the subscribers of the data. Passive updates occur only on subscriber request and usually involves a comparison of local and remote timestamps before transmission. For example, passive updates are typically used to download large volumes of 3D model data. Caching data and comparing their timestamps helps to reduce the need to redundantly download the same data set.

The initial synchronization behavior determines how the local and remote keys should be synchronized when the links are first formed. That is, clients are able to choose to synchronize automatically based on the keys' timestamps. That is the older key will be updated with information from the newer key. However the client may also choose to force synchronization from the local key to the remote key, and vice versa, regardless of timestamp. Of course clients may choose to perform no initial synchronization at all.

Subsequent synchronization behavior specifies the manner in which data is synchronized when local or remote updates to keys occur. The same options as for initial synchronization hold.

The default link property is to use active updates with automatic initial and subsequent synchronization.

Key Properties

Keys may be defined at a client's personal IRB or at a remote IRB provided the client has the necessary permissions. Keys may either be transient or persistent. Persistent keys are keys that will be stored in the IRB's datastore so that when a client or server re-launches, the data will still be retrievable by specifying the same key identifier. Clients determine whether a key is to persist by asking the IRB to perform a commit operation on the data. In addition simple locking functions are provided to allow clients to lock local or remote keys. Locking calls are non-blocking to prevent realtime applications from stalling when attempting to acquire locks on keys. Instead the locking call accepts a user-specified callback function that will be called when a lock has been acquired or when any relevant event pertaining to the lock occurs.

Asynchronous Triggering of Events

Many events may arise during the course of distributing data between clients and servers. The client/server may need to be notified so that appropriate actions may be taken in response to these events. It is inefficient for realtime VR applications to poll for such events. Instead the programs provide the IRBi with callback functions that the IRBi may call when the event arises.

Some examples of events include: new incoming data event; IRB connection broken event; QoS deviation event.

Whoa! Time out! Enough for now!

CAVERNsoft will support many more features in the course of its development, for example, higher level classes will be provided to support avatars, audio/video conferencing, etc.

For now this is enough to get you started on working with CAVERNsoft. The section below gives you a pictorial guide of how to code with CAVERNsoft.


Pictorial Outline of Coding with CAVERNsoft

Assume you are writing a CAVERNsoft client application. The first thing you need to do is initialize CAVERNsoft. This is done by calling CAVERNInit. Once you call init you get a handle to your personal IRB (Information Request Broker). Hang onto this handle since it is the primary means of commanding CAVERNsoft.

Once you have a handle you can define a key with it. A key is just a reference to a place in the IRB where you can store your own information. So once you have a key defined you can start putting data in it.

To share the information in the local key with another remote client you need to first create and open a channel to the remote client (of course the remote client must also be built from the CAVERNsoft library).

By specifying a channel you may choose between reliable and unreliable data delivery; and, soon various other networking characteristics such as quality of service (latency, jitter, bandwidth).

Once the channel is established you can link the local key to a remote key. Now anytime you place new data in your local key the data gets automagically forwarded to the remote key and vice versa. If you wish to store the data in the local or remote key permanently you need to perform a local or remote commit, respectively.

So basically what we've created is a notion of a distributed shared memory with some added database capability. Put in another way it is a database system with active updates. Whatever you want to call it the keys are now intimately linked. The difference between this scheme and other distributed database-like ideas is that the channels over which data is delivered may be tuned to support reliable as well as unreliable networking protocols (like TCP, UDP, Multicast) as well as set up networking Quality of Service features. So although this all feels like a regular distributed shared memory or database it has some slightly unconventional networking nuances. This way we can merge all three concepts simultaneously into one (the idea being to "trick" VR deverlopers into being networking as well as database programmers all at the same time.)

With this you can create multiple channels between clients, and you can choose which pairs of keys are linked over which channels. A channel can support multiple links. Multiple clients may link their keys to a single key on a single central client (server). Proper propagation of data is maintained by CAVERNsoft and the constraints that you gave it when you initially created the link.

Final note about multicasting

Multicast channels in CAVERNsoft create an exception to the general local/remote-key-link concept. Whereas in point-to-point channels, links can be created between a local and a remote key of possibly differing names; when a key is linked over a multicast channel, all keys on remote clients that wish to share this information must first open a multicast channel to the same multicast address AND must also have the same key names. I have thought about a key-aliasing scheme to keep things exactly the same as with links over point-to-point channels but it really defeats the purpose of using multicasting in the first place.

This "brief" introduction should give you enough of the general idea to start writing CAVERNsoft applications. The specifics of the API calls can be found in the API documentation. Also there are a few examples included in the installation that will help get your feet wet.

For annotated example programs take a look here.