Useful Tips


What if I have an existing CAVE program written using the CAVE's networking calls, how do I transition to CAVERNsoft?

Included with CAVERNsoft version 2.5 and above is the CAVE to CAVERNsoft library that mirrors all of the CAVE library's networking calls but implements the underlying networking with CAVERNsoft. This will allow you to extend the networking capabilities using more of CAVERNsoft's services as you feel comfortable. The CAVE2CAVERN library is available in the tools/CAVE2CAVERN directory of the CAVERNsoft distribution.


What is the difference between the SPROC and PTHREAD version?

SPROC refers to SGI's native multithreading library. PTHREAD is the POSIX multithreading standard. Until IRIX 6.5.3 there have always been difficulties with getting PTHREADS to work safely with arena locks that are often used in the CAVE as well as Performer libraries. There may still be additional difficulties that we have not detected yet. For the most part if your program contains no graphics then the PTHREADS version is safe to use. If you have IRIX 6.5.3 the PTHREADS should be safe to use with the CAVE and Performer library. Otherwise it is advisable to use the SPROC version of the library.

There are now new Makefile includes:
CAVERN_MAKEFILE_INCLUDE_PTHREAD_32, CAVERN_MAKEFILE_INCLUDE_PTHREAD_N32, CAVERN_MAkEFILE_INCLUDE_SPROC_32, and CAVERN_MAKEFILE_INCLUDE_SPROC_N32.

Use these instead of the standard CAVERN_MAKEFILE_INCLUDES_32 and CAVERN_MAKEFILE_INCLUDES_N32.

Caveats:

In the SPROC library you have a limit on the maximum number of threads and mutexes/condition variables that can be created. This limit is set a default of 64 threads and  4194304 bytes (4Mb). If your program appears to crash at the creation of a new thread or mutex and you happen to be creating lots of keys, links, channels etc, you may need to increase those default values.

There are 2 ways to do this. Either by setting the following environment variables:
CAVERN_MAX_THREAD, CAVERN_MAX_MUTEX_MEM

or

setting the values in CAVERN_initAttrib_c that is used by CAVERNInit().

Another thing you must remember to do when using the SPROC version is that when you exit your CAVERN program you need to do the proper cleanup. This basically amounts to deleting your CAVERN_irb_c object which will perform a proper shutdown of Globus hence closing all SPROCs. If you simply hit Ctrl C the SPROCs will remain and will prevent you from running another CAVERNsoft program that uses the same network ports.

Eventually Globus will trap for interrupts and do the appropriate cleanup.


Where are the example programs?

The example programs can be found in the demos directory of the CAVERNsoft distribution.  Go to this link for annotated program examples.

My 2 remote CAVERN programs don't seem to connect properly.

If on startup one CAVERN program does not seem to be able to connect to a remote CAVERN program running at a remote site, even though the 2 programs seem to work locally between 2 different computers: it maybe because your local and/or remote machine is not giving CAVERN its full hostname, but instead an abbreviated name. E.g. A full hostname might be ivan.evl.uic.edu. An abbreviated one would be ivan. You can tell if this is the case by running a CAVERN program and looking for a line similar to the following:

CAVERN: CAVERNInit: Listening on Hailing Channel: x-nexus://ivan:10000/

To solve this problem you will need to add the following file to your home directory:

.resource_database

In it you need to place a declaration to identify the domain name of your local machine. (You may also need to do something similar on your remote machine). For example if your local machine is called ivan and the full hostname is actually ivan.evl.uic.edu you need to add the following line to the .resource_database file:

ivan domain=.evl.uic.edu

The next time your run the CAVERN program you should instead see something like this:

CAVERN: CAVERNInit: Listening on Hailing Channel: x-nexus://ivan.evl.uic.edu:10000/

If for some strange reason this still does not appear to work then try the following and then run the program again.

setenv GLOBUS_HOSTNAME ivan.evl.uic.edu


How can I tell CAVERN to use a different IP address for communications rather than what it finds from the domain name server?

Create a file called .resource_database in your home directory. In it place a line similar to the following:

ivan.evl.uic.edu tcp_interface=111.8.94.38 udp_interface=111.8.94.38

This will tell CAVERN that when you try to establish a TCP or a UDP connection to a remote client/server running on ivan.evl.uic.edu it will use the address 111.8.94.38 rather than what the domain name server finds.

This is sometimes necessary if your computer has 2 network interfaces (one may be regular ethernet and the other maybe ATM). The ATM connection may have a different IP address than what the domain name server advertises. So this will let you use the correct IP address.


Can I run 2 CAVERNsoft programs on the same computer?

YES. Usually you do this if you wish to have one CAVERNsoft program connected to another CAVERNsoft program that happens to be running on the same computer (for the sake of convenience during debugging of your program.)

To do this each CAVERNsoft program must start on a different port. There are 2 ways to do this. The quick and dirty way is to simply set the CAVERN_PORT environment variable to the desired port. For example:

setenv CAVERN_PORT 9000
Alternatively you can do it from within the CAVERN program.
Here is an example of initializing 2 CAVERN programs to run on 2 different ports.

Program1:

CAVERN_initAttrib_c initAttr;
initAttr.setPort(7771);
personalIRB = CAVERNInit(&argc,&argv, &initAttr);
Program2:
CAVERN_initAttrib_c initAttr;
initAttr.setPort(7772);
personalIRB = CAVERNInit(&argc, &argv, &initAttr);
Program 1 will initialize CAVERN on port 7771 and program 2 on port 7772.

Now, for example, in order for program 1 to open a connection to program 2, program 1 must do the following:

CAVERN_irbChannel_c *aChannel = personalIRB->createChannel();
CAVERN_irbId_c remoteIRB;
// Insert your own hostname below.
remoteIRB.setAddress("ivan.evl.uic.edu");
// Program 2's port is specified here.
remoteIRB.setPort(7772);
aChannel->open(&remoteIRB, NULL, CAVERN_irbChannel::RELIABLE, &status);

How to take a UNIX file and upload it into a remote CAVERNsoft IRB.

If you wish to upload a UNIX file into a remote IRB (the remote IRB must already be running for this to work) use the program called: The syntax is: where: This program is located in the tools directory of the CAVERNsoft distribution.

How to download data in a remote IRB to a local UNIX file.

If you wish to download the data stored in a key on a remote IRB to a local UNIX file use the program called: The syntax is: where: This program is located in the tools directory of the CAVERNsoft distribution.

Debugging a CAVERNsoft program

It is difficult to debug CAVERNsoft programs with DBX since DBX on the SGI's is not designed to work well with threaded programs. Unfortunately printfs are your main salvation. CAVERNsoft provides a thread-safe printf that you can use to print your own debug messages. Its prototype is:

How to use channels efficiently

As CAVERNsoft allows you to link any number of keys over any given channel there may be some ambiguity over what is the optimum number of channels to create and what is the optimum number of links to assign to each channel. Also there is the issue of what kinds of channels are best for transporting different kinds of data. Here are a few rules of thumb:

For transporting avatar tracker information a separate unreliable channel should be assigned. This channel can host all the tracker traffic for the avatars.

For information that conveys the state of entities in the virtual environment use a separate reliable channel.

For sharing 3D geometries between clients etc, use yet another separate channel. Do not simply combine the latter 2 channels together as large amounts of geometry data can easily slow down smaller state data. By assigning them to separate channels the transmission of the data is interlaced. 


CAVERNsoft Fault Tolerance Provisions

CAVERNsoft was designed with persistence in mind and hence it makes every effort to not crash or terminate a program when channels and links are closed or broken. Instead it offers callbacks to user-specified functions that can be used to alert the program of such events. The program can then proceed to attempt to re-establish the channel. At the current time this must be done manually by the user. In the future, CAVERNsoft will attempt to re-establish connections automatically.

So currently when you receive an event that a channel has broken, simply delete and create a new channel object, delete all the previous link objects and then create all new links with the newly open channel. You do not have to redefine the keys. Those will not be affected. Note however that we can't protect you against your own accidental accesses to dangling memory pointers. Those will surely kill your program however they should not adversely affect the remote IRB programs to which your program may be connected. Your disappearance will eventually be noticed and any resources that you may have held subsequently recycled.

As part of CAVERNsoft's fault tolerance provisions, you may notice that there are some status values returned from the various C++ member functions that state that a link, channel, or key is no longer valid (or stale). These statuses are provided so that you do not accidentally access dangling pointers that point to deleted objects. Everytime you access any of CAVERNsoft's object member functions it verifies that you are accessing a valid object before it proceeds. This allows CAVERNsoft to dynamically tear down broken connections and reclaim system resources without adversely affecting your program.

To reclaim resources CAVERNsoft spawns off a separate garbage collection thread which monitors your personal IRB every 10 seconds. Hence some times when you disconnect from an IRB-based server and then reconnect too soon, it may not allow you to re-create certain links because the old links have not yet been purged from the system. After 10 seconds you can try again and odds are, everything will be fine. Later we will provide a more responsive garbage collection system. 


CAVERNsoft and concurrency

CAVERNsoft is written with Nexus' threads package which are basically macro definitions over pthreads. Hence if you need to, you may create your own threads by using the standard pthreads or Nexus threads calls. If you have never programmed with threads before it may be easier to use CAVERN's thread classes (CAVERNplus_thread_c, CAVERNplus_mutex_c, CAVERNplus_condition_c). These are classes that encapsulate the most basic functionality of Nexus/pthreads.

CAUTIONARY NOTE: You should exercise caution when working with fork() as forking copies of threads can produce unpredicatable results. In general the best policy is to perform all your forks before calling CAVERNInit(). Hence in the CAVE, CAVERNInit should be done after the CAVE has forked off all its draw processes.


Does CAVERN have any ability to lock keys?

Yes. Locking of keys works in a manner similar to RCS. These locks can be transient (held for the runtime life of the IRB) or persistent (stored in the database and hence maintained like RCS). See: CAVERN_irbKey_c::lock(), CAVERN_initAttrib_c::setPassword(), CAVERN_irbLink_c::lockRemote().

For more information look at CAVERN_irbAttrib_c, CAVERN_irbKey_c and CAVERN_irbLink_c classes. Also look at the example: demos/locking.


Does CAVERN have any ability to do network performance monitoring?

Yes. Look at CAVERN_irbAttrib_c and CAVERN_irbChannel_c classes. Also look at the example: demos/performance_monitoring.