Lake Bonney Expedition 2008

Bart and Bill test that the profiler is spooling cable in and out properly. The open profiler electronics pod is sitting on wood blocks between them

Tuesday
December 9, 2008

Thanks to Bart’s long night with the servo driver electronics, we were ready to test the fix this morning. We hooked the pod, with electronics still exposed, to the bot and ran some tests in air to see if everything was functioning normally. Once we saw that the profiler drum was spooling out and winding up as directed we sealed the housing back up and spent a few hours remounting it into the bot. Now it was time for the real test—putting it back in the water. We lowered the bot nervously. Every time we open an electronics housing we run the risk of scratching the pod’s sealing surface or pinching an o-ring. Either mistake will allow the pod to leak, an obvious catastrophe. We were relieved, therefore, when the leak detector board in the pod showed normal data, no leaks. Even better was the news that the profiler was working, even after it had cooled down. We weren’t getting any temperature faults. Another hurdle is out of the way.

Here is the pod we have to remove, right in the center of the vehicle

Bart slowly slides the electronics pod and pulley assembly out of the vehicle

Bill and Bart inspect the pod after we’ve detached the pulley assembly

Bart works late into the night opening up the pod, digging through the layers of electronics and soldering in the resistors

Monday
December 8, 2008

With yesterday’s success and some navigation data for the programmers to occupy themselves with, today seemed like a good day to work on the problem with the profiler. Through some research, Bart had established a few things. For one, the driver’s manual claimed that at the worst case limits, the driver would still give full performance at 0ºC, even though ours had failed at 10ºC. 0ºC is colder than the temperature the driver will be operating at in Bonney, so the driver should have worked. Second, it turns out that the manufacturer had improperly set the low temperature limit, an oversight they apologized for and will repair in all future units. Third, the temperature fault was being triggered by a thermistor in the servo driver electronics. There are several ways to get around this problem and all of them involved getting into the driver’s electronics. What Bart decided to do was to spoof the electronics by adding a fixed value resistor.


The first step of this repair was to remove the housing that contains the profiler electronics. Unfortunately, this is not easy. Of all of the housings we might need to open, the profiler electronics pod is in the physical heart of bot, buried in cables, metal frames, pulleys and so on. We had to remove the drop sonde from its cable, remove the pulley, and undo dozens of nearly-impossible-to-reach bolts. To everyone’s surprise, we had the pod out in only two hours.


Pulling the pod out of the bot was only the beginning of the process. Once we had opened the pod, it took Bart several more hours to meticulously remove the electronic components that blocked his access to the servo driver electronics, where he needed to work. He decided that the easiest fix to our temperature fault problem was to spoof the firmware temperature fault control so that it would never trip off. He did this by soldering a standard value resistor in parallel to the thermistor that had been built into the driver by the manufacturer. A thermistor is a like a temperature sensitive resistor. At lower temperature it provides less resistance and at higher temperature it provides more resistance. Placing another resistor in parallel to the thermistor adds to the total resistance “felt” in that part of the circuit. Since the firmware equates higher resistance to higher temperature, it is fooled by this new high resistance to think that the temperature is higher than it really is and it doesn’t trip the fault.


Its good in theory and we all hope its going to work.

Vickie unspools the kilometer-long fiber optic in preparation for the bot’s 380 meter journey away from the melt hole today

Vickie tracks the bot’s magnetic beacon using a loop antenna

This image comes from the bot’s forward looking camera. When we saw the wrench in the field of view, we knew that we had reached the flagged point on the surface successfully

The bot moves along under the ice, the string is wiggling above the camera is the fiber optic line. It reaches the edge of the melt hole, it identifies the blinking light, locks onto it and ascends to a cheering crowd

Sunday
December 7, 2008

Today we decided to spread our wings a little bit and try a longer run. We had two goals to start with. The first was to make sure the navigation really is rock solid even over long distances and the second was to try and find another piece of lost equipment that Peter thought might be waiting to snag us. These goals would take the bot 380 meters from the melt hole, which is actually the farthest we’ve ever sent the bot away from mission control, even including all of our previous tests in the states and the DepthX missions in Zacatón. In preparation for this rite of robot passage, Vickie swapped out the 400 meter long fiber optic line we had been using up to now for a 1000 m line. While normal fiber optic would sink and droop down to the halocline in Lake Bonney, this fiber optic has a special coating that allows it to float up to the ice ceiling. This way, when the bot spins in a circle or lowers the sonde, it won’t get wrapped up in the fiber.


A destination point was entered into the bot’s navigation system and Endurance motored off while Peter, Bill, and Vickie tracked it from the surface. The target was the last known GPS location of an old ablation stake (essentially an ~8 meter long rod used to measure how much the lake ice ablates over time) that had disappeared into the ice in a previous year. Peter said there was a small chance that the stake could be sticking straight up out of the lake sediment and might serve as a fiber optic snag if we didn’t check it out first.


The GPS point we had turned out to be inaccurate by several meters but there was also a flag marking the spot on the surface so, since we were able to locate the bot from its magnetic beacon, Bill was able to report a heading and distance to Shilpa, Kristof, and Chris at mission control and they were able to drive right to the spot, confirming that both navigation and beacon were working correctly. A wrench securely tied to some rope had been lowered down a small melt hole at the target so that, looking from the forward-looking camera, mission control would be able to confirm that they reached the spot. Reach it we did and we proceeded to investigate the area, spinning the vehicle and using the forward-cam to look for the missing ablation stake. After some time spent hunting for the stake we decided to bring the bot home for the final test of the day: visual homing.


One of the interesting technological problems with working under a 3 meter ice cap is the question of how to get a 2 m x 2 m bot covered with delicate instruments back out of the melt hole you tossed it into. Coming from the DepthX project, we knew that the bot’s dead-reckoning navigation was quite good and could get us back to within a few meters of the melt hole. But short of manually driving the vehicle up the 3 meter tall melt hole, how could we get it to ascend at the right time? The answer was machine vision. We have an upward looking camera on the bot and above the center of the melt hole we have a 12 watt LED light that blinks on and off on a specific frequency. Shilpa worked with Aniket and Greg, two Stone Aerospace programmers back in Austin, to write a visual homing program that would start up when the bot approached the melt hole. It would use the camera to identify all of the light sources above it, pick out the bright light that blinked at the right frequency, lock onto that light, and then kick in the appropriate thrusters to center the vehicle under the light and follow it up to the surface. That is, if it worked.


We had tested the visual homing in our final bot tests over the summer and it appeared to work well in the wide open waters of NBL’s brobdingnagian tank. However, we had never been able to test how it would perform in a tall, narrow tube like the melt hole, where the consequences of running just a little off course meant hitting the wall. Shilpa urged us all to have faith in the program as we leaned over the railings around the melt hole, looking down and waiting for the bot. The orange edge of the bot’s syntactic became visible in the southwest quadrant of the melt hole. It started to pass under the hole, still under normal navigation. The regular pulse of the blinking light flashed off the water surface onto our faces. As the bot passed under the hole it stopped, shimmied a little to one side and then to the other, centering itself before ascending gracefully up the hole. The skin of the syntactic broke the surface of the water and the whole room cheered. Not only had the bot navigated to a target 380 meters away under a 3 meter ice cap, but it came home and, using the visual homing, popped straight up to the surface like all good robots should. We are beginning to feel like all of our hard work is paying off.

Peter displays the mystery experiment pulled out of the lake. The white goop on the body of the thing is salt from the lake. We salvaged the square of Plexiglas for use as a prototyping material later - you never know what you’ll need in the field

Kristof, Shilpa, and Chris track the bot’s progress from mission control

One last shot of the team before Bob and Annika leave. From left to right: Bill, Chris, Vickie, Annika, Bart, Kristof, Shilpa, John, Peter (standing in back), Bob, and Maciek

Saturday
December 6, 2008

At breakfast today, Peter announced that he and John had decided that we would not switch to our second melt hole further down the lake as we had originally planned. Our time this year is better spent collecting more data here near the glacier and running the sonde drops we’ve planned for this side of the lake than to spend the 4 days it would have taken to break down the Bot House and move it to the other end of the lake and set up shop all over again.


We decided to proceed with more non-profiler tests today while Bart continues his dialog with the servo driver tech support. Today is Friday in the states so it his last chance before the weekend to talk to them before we attempt a solution.


Today’s testing schedule included doing some ground truthing of the navigation programs and to test our recovery beacon. Happily, it was easy to both at the same time, since we could use the recovery beacon to locate the bot and verify if it actually navigated to the points it was programmed to go to. We sent the bot down under the ice and gave it some coordinates to drive to about 10 meters away from the Bot House. Bill and Vickie, meanwhile, suited up and went out side with the loop antennas.


Our recovery beacon is a meter-long cylinder that hangs inside the bot frame and produces a magnetic field. The field can be sensed above the ice by a loop antenna, essentially a coil of wire about a half meter in diameter. The loop antenna plugs into a receiver box and the receiver box has a set of headphones. The operator wears the headphones and can hear the signal. When the signal goes null, or makes no noise, then the loop is pointed directly at the robot. When there is a loud tone then the loop is pointed away from the bot. With two people using loop antennas, it is a simple matter to zero in on the beacon.


When mission control radioed that the bot had reached its waypoint and stopped, Vickie and Bill located the bot and marked the spot with an ice screw. A GPS check confirmed that the bot appeared to be navigating correctly. Good news!


The next thing Peter wanted to check on was a mystery object stuck in the lake. Several researchers have worked in Lake Bonney over the years and occasionally the equipment from their various experiments — ropes, sediment traps, stakes, etc--is lost or forgotten in the lake ice. One of our biggest fears is that Endurance will run into one of these objects dangling below the bottom surface of the ice and get the fiber optic snared. When we chose the site for our Bot House, Maciek found some crossed bamboo stakes lying on the ice nearby. It turned out that there was a string tied to them and some sort of object, one of these old experiments, dangling from the string in the water. We navigated to where we calculated the bamboo stakes to be and, approaching carefully, looked at this mystery obstacle through the forward looking camera. We could see some sort of dumbbell - shaped object but couldn’t tell what it was. Maciek then melted the object out of the ice. When Peter returned to the Bot House carrying the retrieved object we all crowded around. It was much smaller than it looked on the screen and no one had any idea what it was, just that it was corroded and crusted with salt that had precipitated onto it from the salty lake. While it remains a mystery, we’re just happy to have this obstacle removed from our path.


While today went well, the end of the day was still a bit of a bummer. Bob and Annika left on a helo flight this afternoon and after a day or two in MacTown they’ll be heading off to holidays in New Zealand. Thanks for your help guys, we’ll miss you!

We put the orange syntactic blocks onto the vehicle to provide flotation

Vickie gets on rope to ballast the vehicle and get the buoyancy just right

The white speckles and fuzz on the bot in this photo are actually thousands of tiny, trouble-making microbubbles.

Friday
December 5, 2008

The first people bringing their breakfasts into our camp mess tent this morning found Bart already on the phone with the servo driver manufacturer. Fortunately, when the comm techs set us up for internet, they also gave us a VOIP phone. After breakfast we tore ourselves away from eavesdropping on Bart’s conversation with tech support to start work out in the Bot House. Just because the profiler is temporarily stalled doesn’t mean we can’t proceed with other tests, so the plan of the day was to put the bot in the water with the syntactic on and take it for its first Antarctic swim.

Before taking the bot down under the ice, we wanted to make sure that the bottom perimeter of the ice hole was snag free. The lake ice is about 3 meters thick and the level of the lake water in the melt hole is about 30 centimeters below the ice surface. The other 2.7 meters of the hole is underwater and this makes it difficult for us to judge if the wall surface is smooth or jagged. To give us a definitive answer, Peter decided to do an ice dive and have a look. As he donned all of the gear associated with ice diving in Antarctica — layers of fleece, a drysuit, dry gloves, and a full face mask — Vickie hooked up his air supply. In these kinds of ice dives, it is nice to keep the scuba tanks in the warm air at the surface and run long hoses from the tanks to the diver’s full face mask. This keeps the tanks and first stage regulators warm, making the dive safer and more enjoyable. The full face mask includes a comm unit so we could talk to Peter during his dive. He took an ice chipper bar down and busted off all of the jagged corners at the bottom of the ice hole.

Once the melt hole was prepared it was time to get the bot ready to swim. We put the syntactic blocks on, lifted the vehicle with the hoist and set it in the water. To make the bot as power-efficient as possible, we always ballast and trim the vehicle. To ballast, we add or remove lead weights to the bot to make it neutrally buoyant, so that without hooks attached it floats just under the surface of the water and neither sinks nor rises. To trim the bot we place the lead weights we’re using for ballast in specific locations on the bot’s frame so that it sits level in the water. The whole process is a little time consuming but a neutrally buoyant and well-trimmed bot won’t use much battery power for thrusting, so it is important to get it just right. In our previous tests in tanks or at the quarry in Austin, it was easy to add and remove lead because you could just lean over the vehicle. Here, however, there is about a meter drop from the top of the platform to the water level, where the vehicle is floating and you would have to lean out horizontally as well, to work with the ballast. Since we didn’t want to drop either a person or our lead weights into the icy water, Vickie rigged a climbing rope to the gantry and, wearing a harness and using climbing ascenders, hung just above the robot to adjust the lead weights more easily.

Once we felt confident that the vehicle was ballasted properly, Kristof, Shilpa and Chris took their seats at the mission control table and drove the vehicle down and under the ice for the first time.

“The ice is smooth!” Shilpa reported.

This was a relief. The top surface of the lake ice is anything but smooth: it is scalloped and cracked with plateaus and valleys and in places can vary in height by half a meter. So naturally we wondered about the underside of the ice. We hoped it would be smooth because it would make navigation easier and it would be less likely for our fiber optic line to become ensnared. We all crowded around the monitor displaying the view of the horizontal-facing camera. Yes, the ice was smooth as glass.

We motored the bot around under the ice testing the various sensors in action. After a while we noticed that some of the sensors that use sonar elements started to drop out and we didn’t know why. We brought the vehicle back to the surface to check it out. As the bot rose up through the hole we noticed that it looked different, kind of…fuzzy. Also, it was too buoyant. At the end of its ascent, the bot was riding almost 2 inches higher in the water than it had when we ballasted it half an hour before. Looking closer we learned that the white “fuzz” we were seeing on the bot was actually thousands and thousands of tiny bubbles coating the entire exterior of the vehicle.

The lake water is supersaturated in dissolved gases and as the bot moved through the water, some of this gas was apparently coming out of solution and forming these microbubbles on the robot’s surfaces. The microbubbles were making the bot significantly more buoyant and were interfering with the acoustics of the sonar transducers. We’ve never had the bot in such gas-rich waters before and didn’t predict this kind of problem.

While we found this surprise to be another puzzle we have to deal with, Peter reminded us all that this is the kind of stuff we are here to learn about. There are two parts to the Endurance mission here in Antarctica: one is to collect data from Lake Bonney and the other is to develop some of the technology that an AUV on Europa would use. In this way, our field work here is proving to be a good analog for a mission on Europa. The exploration of Europa is bound contain some surprises, just like our exploration of Lake Bonney and the microbubbles. The more we can learn about possible issues like these in the experimental phase here on Earth, the better we could design an AUV for alien environments.

Our attempt at a quick fix for the microbubbles on the sonars was to take a small amount of Simple Green hand cleaner and rub it over the sonar faces. This seems to keep the bubbles from forming on these surfaces but we might be able to find a better substance to act as a surfactant here. To counter the extra lift given by the bubbles, for now we just threw an extra pound of lead on the vehicle.

We took the bot back down under the ice for some more tests. For one thing we wanted to test one of our bot recovery procedures. Now that we knew that the underside of the ice was smooth, we figured that if the bot were to die, that is, lose communications with mission control run out of battery power, we could pull it back in by hand by tugging on the fiber optic line we use to communicate with the bot’s cPCI. To make sure, we did a practice run of physically pulling the bot back in with an extra safety cord we had tied on. It worked very easily, so we know that in a crisis we have a way to get the bot back.

During this emergency practice, we had the upward-looking camera on, and were displaying its images on one of the mission control monitors. The upward-looking cam is at the top of the bot and faces directly up. Normally we would only use it for visual homing to help the bot return to the melt hole at the end of a mission. What we didn’t expect is the kind of images we would see while the bot was just running around under the ice. We found that the ice was amazingly clear and that a lot of sunlight came through. There are thousands of bubbles in the ice. Some bubbles contain sediment that was blown onto the ice from the land and frozen in, some bubbles are white and clear. They form columns going from the bottom towards the top. It was beautiful and Peter added enthusiastically that the views from this upward camera were a source of data that no one had recognized before. The ice was clear enough that we could even tell when we had crossed under the Bot House floor.
When we finished the remaining tests for the day we pulled the bot up for the night, put the batteries on charge and went back for dinner.

Kristof, John, and Peter watch intently as the bot is lowered into the melt hole for the first time

John and Peter get the Seabird pump on the drop sonde working

We watch from above as the spooler drum (on the right) unwinds the bright green Ethernet cable that holds the sonde (underwater on the far left)

With the bot back on the surface, Chris plugs directly into the profiler electronics pod to investigate the servo motor malfunction while Kristof looks on

Thursday
December 4, 2008

Today we put the bot in the water. Since the vehicle hasn’t been wet since our last tests at NBL (NASA’s Neutral Buoyancy Lab) in August we wanted to check all of the electronics pods for leaks and check that all of sensors and instruments were working. The bot has traveled halfway around the world, after all. To do these checks we hung the bot from the hoist without the orange syntactic blocks we normally use to make the vehicle float. John and Peter paid particular attention to their scientific instruments on the drop sonde. They adjusted the water pump on the sonde, raising the bot out, priming the pump and lowering it again a few times to watch its behavior. With the pump in order, Chris, Shilpa and Kristof ran through the checklist of sensors and electronics: No pod leaks, Sonars working, DVL (Doppler Velocity Log), depth sensors, IMU (Inertial Measurement Unit), cameras, lights and so on, all working. Check, check, check…

One of the last few tests was to check the science payload package, the profiler. The profiler is made up of several parts. First there is the drop sonde, basically an aluminum frame with all Peter and John's water chemistry instruments strapped onto it. This sonde hangs from the bot by a Kevlar reinforced Ethernet cable. The cable feeds over a pulley and onto a large spool on the opposite side of the vehicle. Servo motors spin the spool to raise and lower the sonde through the different water strata in the lake. Normally, with the floatation blocks in place, we are not able to see much of the spooler functioning, only the up and down motion of the sonde is visible. Since we left blocks off for this test were able to watch the entire profiler in action underwater. It was satisfying to see how smoothly the level winder guided the green Ethernet cable between the spool and the pulley as it wound and unwound, raising and lowering the sonde. Several minutes into the test run, however, the spooler stopped unexpectedly. Chris noted that the software was reporting a temperature fault—it was too cold. What we found was that if we heated the electronics pod for the profiler in air with the Herman Nelson heater, the driver would function correctly but once the pod had cooled to 10ºC in the nearly freezing water, the driver would stop.

Of course we had considered the effects of Antarctic temperatures on the vehicle before. We wondered if such cold water would cause equipment malfunctions. In February we operated the bot under ice in Lake Mendota in Madison, Wisconsin. Those tests identified a problem with the electronics that control the thrusters. We later traced that problem back to some poorly soldered connections—a slip in quality control from the manufacturer. The faulty electronics were immediately replaced and the bot suffered no further problems from the cold. But during the February tests in Wisconsin the profiler was still in the design phase, it didn’t exist and thus couldn’t be cold tested. So what exactly is the problem? Is there a way we can fix it in the software settings, a minimum temperature parameter we can change to allow the servos to run at lower temperature? There isn’t anything about it in the servos’ manual and, given the time difference between here and the states, we’ll have to wait until morning to ask the manufacturer.

Bill finds some hardware to use on a new mounting bracket for the DeltaT sonar

The USBL and HID lights were just a few of the components to attached to the bot today

The team works to get the bot put back together and get the Bot House set up as a workshop / mission control

Wednesday
December 3, 2008

After our brief stint as Antarctic carpenters, we have returned to doing what we do best — working with the robot. Today we got the Bot House set up as a working mission control. The programmers set up their computers along the far wall and boxes of tools, robot batteries and spare parts were unpacked and set up on tables along the two long walls of the building. The bot and gantry dominate our workspace and setting up desk and bench space for 8 programmers and technicians was a little tricky. With everything unpacked and the empty shipping cases stacked high along the walls we set about getting the bot into working order.


First of all, the bot had to be put back together. In Texas, before we shipped the bot to the Ice, we removed the most delicate sensors and packed them in padded cases to be shipped separately. This ensured that they weren’t damaged in transport and it reduced the weight of the remaining vehicle so that it would be light enough to be flown on the helo trip from McMurdo to Lake Bonney. Now that we’re here though, all of these sensors had to go back on before we can think about putting the bot in the water.


As Vickie bolted each sensor back to its proper place, Shilpa and Kristof worked on some of the vehicle code. Chris and Bob assessed the bot’s giant batteries and started charging them up. Bart sorted hardware and worked on developing mission plans. When Vickie went to mount the Delta T sonar to the vehicle she noticed that the bracket that should attach the instrument to the bot wasn’t here. Talking with Bill, they realized that somehow in the rush and confusion of packing the vehicle, tools, spare parts and whatnot in Texas, the bracket they had ordered had never been delivered and thus never brought. Bill and Peter therefore, designed and constructed a new mounting bracket out of some extra prototyping materials we had brought along for just this kind of thing. So far this looks like the only thing that didn’t make it to Antarctica from Texas, not bad!


At the end of the day, most of the vehicle is back together and we should be able to start real testing tomorrow.

Blog entries from previous weeks can be found below:

Week 1 - Nov 26, 2008 - December 2, 2008