New 4 wall CAVE programming interface (CAVElib 2.6alpha3)

(This documentation is for the alpha3 version of the library, found in ~pape/cave/src/v2.6/alpha3/. The features documented here are unchanged from alpha2; the only significant change is that pfCAVE now supports 4 wall operation.)

The new application programming interface for the distributed (i.e. 4 wall) CAVE is based on the idea of "display data", which is pipelined from the computation process to the display processes. The main process requests a special buffer of shared memory; it performs computations with the buffer, and then tells the library to pass on the new data. The data will be sent, frame accurately, to the display processes on the local (master) node, and on all the slave nodes. The application's rendering functions request a pointer to the latest display data buffer, and draw using that data.

The display data buffer can contain either all the shared data needed in rendering, or messages describing changes to the database. The second approach (messages) may be needed if an application has a very large data base and only makes small changes at a time.

This new interface also takes care of access control to the shared data, even in non-distributed CAVE applications. You should not need to do any semaphoring/locking to synchronize the computation and display processes's access to the display data.

The basic approach to using the display data functions is as follows:

	datatype *calcbuf;

	main()
	{
	CAVEConfigure();
	calcbuf = CAVEAllocDisplayData(size);
	if (CAVEDistribMaster())
		init calcbuf
	CAVEDisplay(draw_function,0);
	CAVEInit();
	...
	if (CAVEDistribMaster())
	   while (1)
		{
		compute with calcbuf
		CAVEPassDisplayData(calcbuf,0);
		}
	else
	   while (!CAVESync->Quit) sleep(1);
	}

	draw_function()
	{
	drawbuf = CAVEGetDisplayData(calcbuf,NULL);
	render with drawbuf
	}

As shown above, both the master and slave nodes need to call CAVEAllocDisplayData, but only the master node should perform the calculations and call CAVEPassDisplayData().

Aside: Note that CAVEDisplay (and CAVEInitApplication & CAVEFrameFunction) may now be called before CAVEInit. This can be used to assure that the functions are called the same number of times on each machine, since they will be called starting from the first frame.

Scramnet, TCP, and Hippi distribution protocols are available. Scramnet may be real or simulated - simulated Scramnet uses a shared memory segment which unrelated processes on a single machine can connect to. Use simulated scramnet via the configuration:

	Distribution scramnet
	AppDistribution scramnet
	Scramnet n
	# Shared memory key (for shmget())
	SimScramKey  1700

Function descriptions

boolean CAVEDistribMaster(void): Same as before - returns TRUE on master node, FALSE on any slave nodes. Always returns TRUE when running in non-distributed mode.
volatile void * CAVEAllocDisplayData(size_t size): Allocates shared memory to use for passing data to the display processes, and returns a pointer to one block for the computation process to use. size is the size of the block in bytes. Four blocks of memory are actually allocated - the one that is returned, one for the display processes to use, and two others that are used in staging the data when it is sent by CAVEPassDisplayData(). The blocks are all allocated using CAVEMalloc(), so its shared arena must be large enough to hold the four blocks, plus any other shared data the application will allocate (the size can be set by CAVESetOption()).
When this function is used instead of CAVEAllocDisplayDataByID(), the application must make sure that all the calls occur in one process, in the same order on all machines.
void CAVEPassDisplayData(volatile void *buf,size_t size): Sends data from the buffer buf, which was returned by CAVEAllocDisplayData(), to the display processes. size is the number of bytes to send, starting from the beginning of the buffer; if size is 0, the entire buffer is sent. The data is copied from buf into one of the staging buffers, which will then be returned by CAVEGetDisplayData() on the next frame.
void CAVEPassAllDisplayData(void): Calls CAVEPassDisplayData() for all display data that has been allocated.
volatile void * CAVEGetDisplayData(volatile void *buf,size_t *size): Returns a pointer to the shared buffer with the latest data which has been passed to the display processes. buf can be the pointer returned by CAVEAllocDisplayData() or one subsequently returned by CAVEGetDisplayData(). If size is non-NULL, it will return the number of bytes which were sent by CAVEPassDisplayData().
boolean CAVEDisplayDataChanged(volatile void *buf): Returns TRUE if a new copy of the display data buffer buf has been passed to the display processes since the previous frame. buf can be the pointer returned by CAVEAllocDisplayData() or one subsequently returned by CAVEGetDisplayData().
volatile void * CAVEAllocDisplayDataByID(int id,size_t size)
void CAVEPassDisplayDataByID(int id,size_t size)
volatile void * CAVEGetDisplayDataByID(int id,size_t *size)
boolean CAVEDisplayDataChangedByID(int id): These functions are the same as the corresponding ones above, except that all data buffers are referenced by application-supplied ID numbers. Each buffer allocated must have a unique ID. ID numbers should always be positive integers - CAVEAllocDisplayData() uses negative numbers for its automatically generated IDs.
This approach is necessary to correctly match the buffers between machines, if display data is allocated in multiple processes or in different orders.
boolean CAVEDisplayDataIDExists(int id): Returns TRUE if a display data buffer has been allocated for ID number id, FALSE if not.
int CAVEGetDisplayDataID(void *buf): Returns the ID number associated with the display data buffer buf.
boolean CAVEDistribOpenConnection(int chanID): Opens a communication channel between the master and slave nodes of the distributed CAVE. chanID is an integer identifying the channel; it must be in the range 0 to CAVE_DISTRIB_MAX_CHANNELID (defined in cave.h, currently 31). Returns TRUE if successful, FALSE if the channel could not be opened. Once the channel is opened, it can be used with CAVEDistribRead(), CAVEDistribWrite(), CAVEDistribBarrier(), and CAVEDistribCloseConnection(); these functions should only be called in the same process which opened the channel. Only one process on a CAVE node should open a given channel.
void CAVEDistribCloseConnection(int chanID): Closes a previously opened distributed CAVE communications channel. chanID is the channel ID which was passed to CAVEDistribOpenConnection().
void CAVEDistribWrite(int chanID,void *buf,size_t size): Sends a block of data over a given distributed CAVE channel. chanID is the channel ID which was passed to CAVEDistribOpenConnection(); buf is a pointer to the data to send; size is the number of bytes to send. When called by the master node, this sends a copy of the data to each of the slave nodes; when called by a slave node, this sends the data only to the master.
size_t CAVEDistribRead(int chanID,void *buf,size_t size): Receives the next block of data sent over a distributed CAVE channel by CAVEDistribWrite(). chanID is the channel ID which was passed to CAVEDistribOpenConnection(); buf is a pointer to the buffer to receive the data in; size is maximum number of bytes to receive. The return value is the number of bytes that were actually read. This function blocks until data is received. On the master node, a single call to CAVEDistribRead() will receive data from only one of the slave nodes, in no particular order; it should be called once for each of the slaves (assuming they all call CAVEDistribWrite()); the number of slave nodes is CAVEDistribNumNodes()-1.
void CAVEDistribBarrier(int chanID): Blocks until all of the distributed CAVE nodes reach the barrier. chanID is the channel ID which was passed to CAVEDistribOpenConnection().
int CAVEDistribNumNodes(void): Returns the total number of nodes in the distributed CAVE.

Performance

This section gives the results of some basic performance tests I have done to see how quickly data can be passed. The program used did no computations (and did not call sginap() or getbutton()); it just spent all its time calling CAVEPassDisplayData() on a single buffer of the given size (64 / 512 / 2048 / 8192 / 32768 bytes). 'AppDistrib hippi' used raw Hippi for the communications; 'AppDistrib tcp (hippi)' used TCP communication over Hippi, by specifying hip-zbox as the DistribTCPMaster. 'AppDistrib tcp (atm)' used TCP communication over ATM, specifying zbox-atm as the DistribTCPMaster.

64 byte data:
   AppDistrib scramnet:     3300 sends/sec			  211.2 KB/s
   AppDistrib hippi:        132 sends/sec			    8.4
   AppDistrib tcp (ether):  360 sends/sec			   23.0
   AppDistrib tcp (hippi):  333 sends/sec			   21.3
   AppDistrib tcp (atm):    330 sends/sec			   21.1
   No distribution:	    385000 sends/sec			24640.0

512 byte data:
   AppDistrib scramnet:     1240 sends/sec			  634.9
   AppDistrib hippi:        132 sends/sec			    8.4
   AppDistrib tcp (ether):  317 sends/sec			  162.3
   AppDistrib tcp (hippi):  330 sends/sec			  169.0
   AppDistrib tcp (atm):    315 sends/sec			  161.3
   No distribution:	    192000 sends/sec			98304.0

2048 byte data:
   AppDistrib scramnet:     380 sends/sec			  778.2
   AppDistrib hippi:        118 sends/sec			  241.7
   AppDistrib tcp (ether):  196 sends/sec			  401.4
   AppDistrib tcp (hippi):  310 sends/sec			  634.9
   AppDistrib tcp (atm):    287 sends/sec			  587.8
   No distribution:	    70000 sends/sec			143360.0

8192 byte data:
   AppDistrib scramnet:     101 sends/sec			  827.4
   AppDistrib hippi:        106 sends/sec			  868.4
   AppDistrib tcp (ether):  90 sends/sec			  737.3
   AppDistrib tcp (hippi):  254 sends/sec			 2080.8
   AppDistrib tcp (atm):    195 sends/sec			 1597.4
   No distribution:	    12500 sends/sec			102400.0

32768 byte data:
   AppDistrib scramnet:     25 sends/sec			  819.2
   AppDistrib hippi:        100 sends/sec			 3276.8
   AppDistrib tcp (ether):  22 sends/sec			  720.9
   AppDistrib tcp (hippi):  147 sends/sec			 4816.9
   AppDistrib tcp (atm):    111 sends/sec			 3637.2
   No distribution:	    2070 sends/sec			67829.8


Notes: Distribution scramnet
       ran between zbox & wall1, no display or tracking processes
       main process using isolated CPU, doing no calculations or getbutton()s

Sample program (bouncing balls)

#include <cave.h>
#include <malloc.h>
#include <gl/sphere.h>

struct _balldata
	{
	float y;
	};

void init_gl(void),draw_balls(void);

 struct _balldata *ball;

main(int argc,char **argv)
{
 CAVEConfigure(&argc,argv,NULL);
/****** Allocate buffer for data shared with display processes ******/
 ball = (struct _balldata *) CAVEAllocDisplayData(2*sizeof(struct _balldata));
/****** Initialize data ******/
 if (CAVEDistribMaster())
	ball[0].y = ball[1].y = 0;
 CAVEInitApplication(init_gl,0);
 CAVEDisplay(draw_balls,0);
 CAVEInit();
 if (CAVEDistribMaster())
	while (!getbutton(ESCKEY))
		{
		float t = CAVEGetTime();
		ball[0].y = fabs(sin(t)) * 6 + 1;
		ball[1].y = fabs(sin(t*1.2)) * 4 + 1;
/****** Pass the new data to the display processes ******/
		CAVEPassDisplayData(ball,0);
		sginap(2);
		}
 else
/****** CAVElib will set CAVESync->Quit when the master calls CAVEExit() *****/
	while (!CAVESync->Quit)
		sginap(25);
 CAVEExit();
}

void init_gl(void)
{
 float redMaterial[] = { DIFFUSE, 1, 0, 0, LMNULL };
 float blueMaterial[] = { DIFFUSE, 0, 0, 1, LMNULL };
 lmdef(DEFLMODEL,1,0,NULL);
 lmbind(LMODEL,1);
 lmdef(DEFLIGHT,1,0,NULL);
 lmbind(LIGHT0,1);
 lmdef(DEFMATERIAL,1,0,redMaterial);
 lmdef(DEFMATERIAL,2,0,blueMaterial);
}

void draw_balls(void)
{
 float sphereParam0[] = { 2, 4, -5, 1}, sphereParam1[] = { -2, 4, -5, 1};

/****** Get pointer to the most recent display data ******/
 ball = (struct _balldata *) CAVEGetDisplayData(ball,NULL);

 czclear(0,getgdesc(GD_ZMAX));

 lmbind(MATERIAL,1);
 sphereParam0[1] = ball[0].y;
 sphdraw(sphereParam0);

 lmbind(MATERIAL,2);
 sphereParam1[1] = ball[1].y;
 sphdraw(sphereParam1);
}