New 4 wall CAVE programming interface (CAVElib 2.6alpha2)
(This documentation is for the alpha2 version
of the library, found in ~pape/cave/src/v2.6/alpha2/.
Documentation for the previous alpha release is here)
The new application programming interface for the distributed (i.e. 4 wall)
CAVE is based on the idea of "display data", which is pipelined from the
computation process to the display processes. The main process requests
a special buffer of shared memory; it performs computations with the buffer,
and then tells the library to pass on the new data.
The data will be sent, frame accurately, to the display processes on the
local (master) node, and on all the slave nodes.
The application's rendering functions request a pointer to the latest
display data buffer, and draw using that data.
The display data buffer can contain either all the shared data needed
in rendering, or messages describing changes to the database. The
second approach (messages) may be needed if an application has a
very large data base and only makes small changes at a time.
This new interface also takes care of access control to the shared
data, even in non-distributed CAVE applications. You should not need
to do any semaphoring/locking to synchronize the computation and display
processes's access to the display data.
The basic approach to using the display data functions is as follows:
datatype *calcbuf;
main()
{
CAVEConfigure();
calcbuf = CAVEAllocDisplayData(size);
if (CAVEDistribMaster())
init calcbuf
CAVEDisplay(draw_function,0);
CAVEInit();
...
if (CAVEDistribMaster())
while (1)
{
compute with calcbuf
CAVEPassDisplayData(calcbuf,0);
}
else
while (!CAVESync->Quit) sleep(1);
}
draw_function()
{
drawbuf = CAVEGetDisplayData(calcbuf,NULL);
render with drawbuf
}
As shown above, both the master and slave nodes need to call CAVEAllocDisplayData,
but only the master node should perform the calculations and call
CAVEPassDisplayData().
Aside: Note that CAVEDisplay (and CAVEInitApplication & CAVEFrameFunction)
may now be called before CAVEInit. This can be used to assure that the
functions are called the same number of times on each machine, since they
will be called starting from the first frame.
Scramnet, TCP, and Hippi distribution protocols are available.
Scramnet may be real or simulated - simulated Scramnet uses a shared memory
segment which unrelated processes on a single machine can connect to.
Use simulated scramnet via the configuration:
Distribution scramnet
AppDistribution scramnet
Scramnet n
# Shared memory key (for shmget())
SimScramKey 1700
Function descriptions
- boolean CAVEDistribMaster(void)
- Same as before - returns TRUE on master node, FALSE on any slave nodes.
Always returns TRUE when running in non-distributed mode.
- volatile void * CAVEAllocDisplayData(size_t size)
- Allocates shared memory to use for passing data to the
display processes, and returns a pointer to one block for the computation
process to use. size is the size of the block in bytes.
Four blocks of memory are actually allocated - the one that is returned,
one for the display processes to use, and two others that are used in
staging the data when it is sent by CAVEPassDisplayData().
The blocks are all allocated using CAVEMalloc(), so its shared
arena must be large enough to hold the four blocks, plus any other
shared data the application will allocate (the size can be set by
CAVESetOption()).
When this function is used instead of CAVEAllocDisplayDataByID(),
the application must make sure that all the calls occur in one process,
in the same order on all machines.
- void CAVEPassDisplayData(volatile void *buf,size_t size)
- Sends data from the buffer buf, which was returned
by CAVEAllocDisplayData(),
to the display processes. size is the number of bytes to send,
starting from the beginning of the buffer; if size is 0, the
entire buffer is sent. The data is copied from buf
into one of the staging buffers, which will then be returned by
CAVEGetDisplayData() on the next frame.
- void CAVEPassAllDisplayData(void)
- Calls CAVEPassDisplayData() for all display data that has been
allocated.
- volatile void * CAVEGetDisplayData(volatile void *buf,size_t *size)
- Returns a pointer to the shared buffer with the latest data which
has been passed to the display processes. buf can be the pointer
returned by CAVEAllocDisplayData() or one subsequently returned
by CAVEGetDisplayData().
If size is non-NULL,
it will return the number of bytes which were sent by CAVEPassDisplayData().
- boolean CAVEDisplayDataChanged(volatile void *buf)
- Returns TRUE if a new copy of the display data buffer buf
has been passed to the display processes
since the previous frame.
buf can be the pointer returned by CAVEAllocDisplayData()
or one subsequently returned by CAVEGetDisplayData().
- volatile void * CAVEAllocDisplayDataByID(int id,size_t size)
- void CAVEPassDisplayDataByID(int id,size_t size)
- volatile void * CAVEGetDisplayDataByID(int id,size_t *size)
- boolean CAVEDisplayDataChangedByID(int id)
- These functions are the same as the corresponding ones above, except
that all data buffers are referenced by application-supplied ID numbers.
Each buffer allocated must have a unique ID.
ID numbers should always be positive integers - CAVEAllocDisplayData()
uses negative numbers for its automatically generated IDs.
This approach is necessary to correctly match the buffers between machines,
if display data is allocated in multiple processes or in different orders.
- boolean CAVEDisplayDataIDExists(int id)
- Returns TRUE if a display data buffer has been allocated for ID number
id, FALSE if not.
- int CAVEGetDisplayDataID(void *buf)
- Returns the ID number associated with the display data buffer buf.
- boolean CAVEDistribOpenConnection(int chanID)
- Opens a communication channel between the master and slave nodes of
the distributed CAVE. chanID is an integer identifying the channel;
it must be in the range 0 to CAVE_DISTRIB_MAX_CHANNELID (defined in cave.h,
currently 31). Returns TRUE if successful, FALSE if the channel could not
be opened. Once the channel is opened, it can be used with CAVEDistribRead(),
CAVEDistribWrite(), CAVEDistribBarrier(), and
CAVEDistribCloseConnection(); these functions should only be called
in the same process which opened the channel. Only one process on a CAVE
node should open a given channel.
- void CAVEDistribCloseConnection(int chanID)
- Closes a previously opened distributed CAVE communications channel.
chanID is the channel ID which was passed to CAVEDistribOpenConnection().
- void CAVEDistribWrite(int chanID,void *buf,size_t size)
- Sends a block of data over a given distributed CAVE channel.
chanID is the channel ID which was passed to CAVEDistribOpenConnection();
buf is a pointer to the data to send; size is the number of
bytes to send. When called by the master node, this sends a copy of the
data to each of the slave nodes; when called by a slave node, this sends
the data only to the master.
- size_t CAVEDistribRead(int chanID,void *buf,size_t size)
- Receives the next block of data sent over a distributed CAVE channel by
CAVEDistribWrite().
chanID is the channel ID which was passed to CAVEDistribOpenConnection();
buf is a pointer to the buffer to receive the data in; size is
maximum number of bytes to receive. The return value is the number of bytes
that were actually read. This function blocks until data is received.
On the master node, a single call to CAVEDistribRead() will receive
data from only one of the slave nodes, in no particular order; it should be
called once for each of the slaves (assuming they all call CAVEDistribWrite());
the number of slave nodes is CAVEDistribNumNodes()-1.
- void CAVEDistribBarrier(int chanID)
- Blocks until all of the distributed CAVE nodes reach the barrier.
chanID is the channel ID which was passed to CAVEDistribOpenConnection().
- int CAVEDistribNumNodes(void)
- Returns the total number of nodes in the distributed CAVE.
Changes from alpha1
- Added function CAVEPassAllDisplayData()
- CAVEInit() now automatically calls CAVEPassAllDisplayData() when starting,
rather than requiring the application to send everything during initialization.
- CAVEPassDisplayData() may be called by slave nodes, but only if
AppDistribution is disabled.
- CAVEMalloc() uses an arena attached at a fixed address. This means
that pointers to CAVEMalloc'ed data can be shared between machines
(assuming the same sequence of CAVEMalloc calls is made on each machine,
of course).
- Other changes, not relating to display data:
- Added functions CAVEGetFrameNumber() and CAVEDisplaySync().
- OpenGL windows are now created borderless, without requiring
changes to .Xdefaults .
Performance
This section gives the results of some basic performance tests I have done
to see how quickly data can be passed. The program used did no computations
(and did not call sginap() or getbutton()); it just
spent all its time calling CAVEPassDisplayData() on a single buffer
of the given size (64 / 512 / 2048 / 8192 / 32768 bytes).
'AppDistrib hippi' used raw Hippi for the communications;
'AppDistrib tcp (hippi)' used TCP communication over Hippi, by specifying
hip-zbox as the DistribTCPMaster.
'AppDistrib tcp (atm)' used TCP communication over ATM, specifying
zbox-atm as the DistribTCPMaster.
64 byte data:
AppDistrib scramnet: 3300 sends/sec 211.2 KB/s
AppDistrib hippi: 132 sends/sec 8.4
AppDistrib tcp (ether): 360 sends/sec 23.0
AppDistrib tcp (hippi): 333 sends/sec 21.3
AppDistrib tcp (atm): 330 sends/sec 21.1
No distribution: 385000 sends/sec 24640.0
512 byte data:
AppDistrib scramnet: 1240 sends/sec 634.9
AppDistrib hippi: 132 sends/sec 8.4
AppDistrib tcp (ether): 317 sends/sec 162.3
AppDistrib tcp (hippi): 330 sends/sec 169.0
AppDistrib tcp (atm): 315 sends/sec 161.3
No distribution: 192000 sends/sec 98304.0
2048 byte data:
AppDistrib scramnet: 380 sends/sec 778.2
AppDistrib hippi: 118 sends/sec 241.7
AppDistrib tcp (ether): 196 sends/sec 401.4
AppDistrib tcp (hippi): 310 sends/sec 634.9
AppDistrib tcp (atm): 287 sends/sec 587.8
No distribution: 70000 sends/sec 143360.0
8192 byte data:
AppDistrib scramnet: 101 sends/sec 827.4
AppDistrib hippi: 106 sends/sec 868.4
AppDistrib tcp (ether): 90 sends/sec 737.3
AppDistrib tcp (hippi): 254 sends/sec 2080.8
AppDistrib tcp (atm): 195 sends/sec 1597.4
No distribution: 12500 sends/sec 102400.0
32768 byte data:
AppDistrib scramnet: 25 sends/sec 819.2
AppDistrib hippi: 100 sends/sec 3276.8
AppDistrib tcp (ether): 22 sends/sec 720.9
AppDistrib tcp (hippi): 147 sends/sec 4816.9
AppDistrib tcp (atm): 111 sends/sec 3637.2
No distribution: 2070 sends/sec 67829.8
Notes: Distribution scramnet
ran between zbox & wall1, no display or tracking processes
main process using isolated CPU, doing no calculations or getbutton()s
Sample program (bouncing balls)
#include <cave.h>
#include <malloc.h>
#include <gl/sphere.h>
struct _balldata
{
float y;
};
void init_gl(void),draw_balls(void);
struct _balldata *ball;
main(int argc,char **argv)
{
CAVEConfigure(&argc,argv,NULL);
/****** Allocate buffer for data shared with display processes ******/
ball = (struct _balldata *) CAVEAllocDisplayData(2*sizeof(struct _balldata));
/****** Initialize data ******/
if (CAVEDistribMaster())
ball[0].y = ball[1].y = 0;
CAVEInitApplication(init_gl,0);
CAVEDisplay(draw_balls,0);
CAVEInit();
if (CAVEDistribMaster())
while (!getbutton(ESCKEY))
{
float t = CAVEGetTime();
ball[0].y = fabs(sin(t)) * 6 + 1;
ball[1].y = fabs(sin(t*1.2)) * 4 + 1;
/****** Pass the new data to the display processes ******/
CAVEPassDisplayData(ball,0);
sginap(2);
}
else
/****** CAVElib will set CAVESync->Quit when the master calls CAVEExit() *****/
while (!CAVESync->Quit)
sginap(25);
CAVEExit();
}
void init_gl(void)
{
float redMaterial[] = { DIFFUSE, 1, 0, 0, LMNULL };
float blueMaterial[] = { DIFFUSE, 0, 0, 1, LMNULL };
lmdef(DEFLMODEL,1,0,NULL);
lmbind(LMODEL,1);
lmdef(DEFLIGHT,1,0,NULL);
lmbind(LIGHT0,1);
lmdef(DEFMATERIAL,1,0,redMaterial);
lmdef(DEFMATERIAL,2,0,blueMaterial);
}
void draw_balls(void)
{
float sphereParam0[] = { 2, 4, -5, 1}, sphereParam1[] = { -2, 4, -5, 1};
/****** Get pointer to the most recent display data ******/
ball = (struct _balldata *) CAVEGetDisplayData(ball,NULL);
czclear(0,getgdesc(GD_ZMAX));
lmbind(MATERIAL,1);
sphereParam0[1] = ball[0].y;
sphdraw(sphereParam0);
lmbind(MATERIAL,2);
sphereParam1[1] = ball[1].y;
sphdraw(sphereParam1);
}