Lake Bonney Expedition 2009

Rachel completes the swap out of the #2 thruster for maintenance.

Marking the furthest exploratory mission for ENDURANCE, Vickie tracks the bot at 1.5 kilometers from the melt hole. Behind her is the "narrows" between west and east Lake Bonney, looking into East Lobe. Scott's team took measurements here December 17, 1903.

The grapel storm turned into a blizzard, obscuring the lab from 500 meters out in a white out.

Vickie gives Rachel a break from fiber tending following the 6 hour mission.

Shilpa, Bart, and Chris watch the new situational awareness visualizer at mission control as the bot "discovers" the melt hole on return from its long trip.

Status of all sonde casts (green = completed) as of November 14, 2009.

Saturday
November 14, 2009

We awoke to news that McMurdo was in Condition 1 (no travel). Here in the valleys it snowed during the evening at the higher peak levels but down low it was still dry.


During the routine (but extensive) pre-mission checklist we found that one of the six thrusters was showing a low level of pressure balance volume so it was taken out for maintenance and a spare swapped in.


Prior to yesterday we had planned to incrementally (and conservatively) work our way out towards the "narrows" separating east and west Lake Bonney some 1.5 kilometers east of the lab. Using the revised mission planner and the improved battery behavior we realized we could acquire all of the remaining east end sonde casts in one mission. A year ago we had felt it would be required to have a secondary melt hole near the east end for "inflight refueling" and had re-designed the charging system on the vehicle for that purpose. Now with the improved vehicle travel efficiency and double power onboard it appeared we could run significant east end forays.


Our plan for today was to conduct sonde casts at E17, E18, E19, D21, E20, F19, F18, F17, E16, E15, E14, F14, and F6. Total mission length would be 3.2 km.


The mission began at 3:52 pm and the vehicle returned to the melt hole at 10:30pm with all sonde points acquired. Importantly, we had conducted a detailed multi-beam side-looking survey at two locations at western head of the Narrows. Multibeam was able to image all the way to shore on both sides. Depth at lake center in this area was 11m under keel, leaving enough to consider a more advanced mission in which the bot actually drove through the narrows and into East Lobe. At D21 (our furthest point today) we were just out of sonar range to image the location measured by Scott.


Shortly after turn around (although still with more than half the sonde casts remaining to be performed) we had our only technical glitch for the day ? at F19 communications dropped out for an unnerving 30 seconds. Then it suddenly came back. Shilpa analyzed the data and began searching a section of the onboard code she had been suspicious of for some time, ever since the earlier data link drop outs. With nearly 100,000 lines of software running the bot this would have been an insurmountable task during a mission, but Shilpa had spent a lot of time tracking this over the last 10 days and now found it - a "memory leak" in a routine designed to track the status of certain vehicle systems. Unchecked on a long mission, the errant process filled up memory and died. When restarted, it freed up memory and the system came back. The fix, until a more optimized re-write of that section could be performed, was to pre-empt and periodically re-boot that process - one of scores running the vehicle. Given the location of the bot at the time (1.5 kilometers from the hole) this was an impressive bit of thinking under pressure on the part of Shilpa.


At 8pm, still more than a kilometer from the lab, the grapel (1mm diameter micro hail) storm began with persistent 25 knot up-valley winds. The lab was already invisible from station F14 and we joked about having to navigate the tracking team back to the lab by following the bot home under the ice. By the time the tracking team returned there had been an accumulation of 1 inch of snow by completion of mission.


By 11pm the bot was on its sled at the lab. Power reserve was 21% when the it reached the melt hole, following a total of 7 hours 3 minutes in the water. At end of mission we found the bot very positively buoyant - adding the 8 kg dive bag we use to lower it each day would not get it underwater. Heavy micro-bubble formation had occurred all over the bot. There were also micro-bubbles all over the sonde (down-looking) camera window.


We drove home to East Lobe camp at 1am.

The ambitious 3.2 kilometer sonde mission planned for November 13, including 19 sonde casts.

Status of all sonde casts (green = completed) as of November 13, 2009.

Friday
November 13, 2009

The lead team of Vickie, Kristof, and Bill was up at 6:40am and off to the bot garage by 8am. They reloaded both battery stacks into their housings and into the bot, then reconnected power lines and re-installed ballast lead. By 10am the system was powered up and showing 100% power charge.


By noon the bot was over the melt hole for the sonde instrument calibration. Outside, winds began to pick up steadily from South Pole (katabatics) to 30 knots gusting to 35 knots. Wind gusts hitting the north end of the lab were violently shaking it - every time it moved it blew out the heater. The air temperature at the start of the mission was a warm 32F but dropping.


We shortly received a radio briefing that helicopters were grounded at McMurdo for at least two days, preventing arrival of chief project scientist Peter Doran. At 12:20pm we performed a sonde test drop to 18m depth to verify changes made by Chris and John to the sonde instrument parameters - all looked good on the live sonde monitor at mission control so we proceeded. By 12:30pm the bot was down at 5 meters depth and performing a test of ice picking behavior. This time, due to variations in temperature, the vehicle was proving to be too heavy so we recalled it to the melt hole and removed 500 grams of ballast.


The wind was blowing down valley strongly for the entire mission without let up. Going eastward the tracking team was being forcefully pushed ahead; coming back was a fight. Taking tracking beacon readings were difficult -- the wind was blowing the coils and the wind drone made it difficult to hear the tone.


The mission was successfully concluded at 7:15pm with an automated ascent up through the melt hole and a 25% remaining power reserve. We had completed 20 sonde drops today: G12, G13, F14, F15, F16, G16, G15, G14, H13, H12, H11, F13, F12, F11, F9, E7, D7, D6, E6, F6.


The total mission length was 3400 meters, including additional post-return navigation testing that was performed in the vicinity of the melt hole involving the USBL secondary localizer. Following the additional navigation tests the power was down to 12% and still usable, validating Bart's hunch of several days ago regarding the battery balancing.


The team returned to camp at 9:20pm (an early night for us) and discovered to everyone's dismay that the winds (which were still driving hard) had uprooted Shilpa?s and Chris's tents. After dinner the entire team broke out replacement tents and helped set new sites up, these tied down to bigger rocks.

Vickie connects a local computer interface to the Profiler housing. Using this and an external laptop computer we were able to respool the instrument sonde.

Chris checks voltages on a power supply terminal strip in the main computer processor housing.

The suspect power terminal block.

With refurbished connections, Chris successfully logs into the bot through the wireless interface.

Rachel reassembles the main computer. This is actually a stack of several processors. Overall there are 45 computer processors in ENDURANCE.

Thursday
November 12, 2009

November 12, 2009 The mission today was clear: find the cause of the communications system failure of yesterday. We had already tested all the elements of the external system yesterday (fiber, data converters) and the internal data hub itself proved to be functioning correctly when run off an external power supply dialed to the appropriate drive voltage. Similarly, the onboard power supplies were also working correctly. It took eight hours to methodically go down the list of likely contenders. In the end, the theory that wire terminations in a screw-type termination block had oxidized and thereby reduced the voltage getting to the communications hub had proved correct. In effect there had been a power "brown out" that was enough to drop the hub for the remainder of the mission after the initial failure at G11. There were parallel communications (via internet) going on with several members of the SAS engineering team in the U.S. and elsewhere to confer on this subject. Ultimately, we pulled the wires leading to the data hub, trimmed them, reterminated and soldered them and then re-installed into the terminal block. The system powered up normally. The general conjecture was that the bot had sat in the MEC engineering center in McMurdo station for 10 months between the last power up and that this had given time for a loose connection to oxidize enough to manifest itself as a reduced voltage coming out of the power supplies. We pre-empted at this point and re-terminated all the other wires going into this same power block.


Meanwhile, Bart used the down time opportunity to play out a hunch regarding the 20% "stranded" power in the batteries. the new design, completed just in time for this season, used two stacks of batteries in series along with custom circuitry to balance the voltages. But that circuitry had to get its power somewhere and it was connected to the first half of the stack, slowly causing the voltage there to drop below the other stack. Safeguard circuitry looked for the lowest cell in the system and if it fell below a certain value (for fire safety) it disconnected the battery. What Bart discovered was a significantly larger power draw by the monitoring circuitry than was needed to run it. So a modification was made to the boards to reduce this demand by a factor of ten. Some test calculations indicated that at the new level of power draw from the monitoring circuits that the overall voltage balancing system would succeed (it was not designed for differential power draw). With the two halves of each of the power stacks (the vehicle uses two sets of two stacks) now balanced (manually) and the control circuitry installed, we anticipated a significant change in available power. Tomorrow's mission will tell.

Vickie "flakes out" the 2 kilometer data fiber spool so that it will deploy without hockling (a tendency of coiled ropes, fibers, and cables to form loops)

Bill operates an instrument for measuring the surface albedo at station F10. These data will be used to cross correlate PAR (photosynthetic active radiation) readings from the vehicle sonde sensors with surface illumination.

Vickie grabs a quick bite of lunch while waiting at station G11 for a decision from Mission Control (450 meters away towards the glacier) on the status of the vehicle.

Wednesday
November 11, 2009

Today was to be the first long range mission, going 300 meters further east than we had reached in 2008. In preparation for this we re-spooled a new 2 kilometer data fiber we had commissioned for 2009. This, plus mission planning and software upgrades keep everyone busy until 2:45pm when the mission got fully underway, targeted at achieving 19 sonde drops. We had sunny weather with a persistent 10 knot wind coming up the valley from McMurdo Sound.


We had already logged the first sonde cast (F10) and were localizing the vehicle at G11 when at 4:45PM mission control called to indicate communications went down on the bot. The team at mission control tried rebooting; changing the short fiber from the main targeting computer; changing the fiber optic to Ethernet conversion boxes, but no go. They then began pulling fiber in to test if communications would come back - due to a kink straightening out - but also no go. So Bart and Kristof began pulling it back using the fiber with a slow, steady 2 kilogram force - enough to gently get it moving under the ice cap. Vickie and Bill tracked it for 50m to assure it was pulling straight back towards the bot house before packing up and returning the remaining 400 meters on foot. The vehicle was in the middle of a sonde caste when the communications had gone dead so it was unclear whether the sonde mission completed and the instrument package homed to the bot.


The vehicle successfully returned to the melt hole at 7:44pm, whereupon an extensive investigation began that would last another day. It was floating neutrally in the water, an important sign that there had been no housing leaks. We made a direct fiber optic connection (bypassing the 2 kilometer spool) but still no connection. Similarly, a wireless link that was activated when the bot was out of the water also would not work. We discovered when the bot was hoisted out of the water that the sonde had not reeled back in - indicating a significant communications problem had affected all systems at once. Vickie connected a diagnostic serial communications cable directly into the Profiler housing and Chris was able to verify that the bot had significant reserve power and was responsive to direct local commands - and we thus respooled the sonde - about 26 meters had been paid out at the time of failure. Further investigations through other independent instrumentation data ports showed that the main processor stack data router was down. We were able to talk to the cameras and several other key instruments but it was the main processor stack that issued commands to operate the various parts of the vehicle. The batteries had more than 50% power remaining. An independent test of the 2 kilometer fiber showed that to be operating as expected.


At 9pm we recycled vehicle power and the wireless link came up temporarily then crashed. We then opened up the main processor housing. There were no "smoking guns" that would suggest a shock-induced failure (e.g. a loose connector from impacting the underside of the ice during ice picking). We held a team meeting to discuss and the general consensus was that oxidation on wires from internal power supply to router may be the problem - will investigate tomorrow.

John Priscu collects fallen blocks of ice for our drinking water from Taylor glacier.

Kristof brings and empty wastewater barrel to the Jamesway hut. Everything from dish and washwater to urine goes into these barrels.

Emma Steger prepares a 7 barrel sling load of wastewater to "retro" to McMurdo.

Tuesday
November 10, 2009

This was a day off. Most people were up at 1:20pm, after having gone to bed around 3am after downloading all the data and cleaning up from the late mission of November 9. Most people wrote blog entries and sorted photos. Kristof and John Priscu collected "glacier berries" - small chunks of ice that had broken off Taylor glacier - and hauled them back to camp in a sled behind their ATV. This has been our main water source.


Today was also a whirlwind of helo activity here with carpenters coming in from McMurdo to work on basecamp improvements and PIs ("Put Ins") of empty wastewater barrels and removal of full ones. Items to be "retro'd" (returned to McMurdo) had to be loaded into cargo nets for sling loading beneath a helicopter. A typical load is seven 55 gallon drums or a similar number of loaded propane tanks. A total of four flights landed here today.

View of the lake bottom at station H4 showing microbial mats.

View of the lake bottom at station H8 showing a strange reddish deposit - probably iron leaching out of the mud.

In Mission Control, Kristof monitors the live video feed coming from the sonde camera.

Sonde cast progress as of November 9, 2009 (green circles).

Frontice precipice about halfway up Mount JJ Thompson at 1am on our way home. The lighting angles in Taylor Valley are consistently striking and distinctive - since we have 24 hour daylight here the only way to tell the time of day without a watch is to observe where the shadows are and what peaks are lit.

Monday
November 9, 2009

The bot was in the water at 11:30am. Around an hour later it was apparent that there was a navigation error which was being manifested as a scaling and rotation error. We subsequently drove the vehicle to stations F5, F4, and F3 which showed a straight line at about 95% scale and rotated 8 degrees counterclockwise from the accurate data from two days ago. We returned the bot to the lab and GPS-located the achieved stations targeted at F5, F4, and F3.


Kristof soon realized that the navigation correction transform obtained from the 2009 calibration had been accidentally applied to the previous 2008 corrections so we had been doubly correcting. We made the correction but decided to put the vehicle on charge for an hour to make up for the lost power before re-starting today's mission. We were back in the water at 3:30pm.


Ultimately this was a 19 point sonde mission. We started off swiftly with F5, G5, G4, G3, H4, H5 then encounterd a data fiber snag when returning from the closest approach to the glacier - probably from the same type of pressure cracks and bulges that had plagued the previous mission. We modified the mission to go to G6 whereupon the fiber pulled free. The remaining stations - H6, G7, H7, H8, H9, H10, G10, G9, G8, F8, F7, F6 - proceeded without issue.

Chris (left), Bart, Shilpa, and Kristof discuss the chronology of the previous day's mission. The team holds its meetings standing up - so everyone gets to the point of their thinking swiftly. We hold these meetings as needed, mainly in the morning to define the mission objectives and roles, but as needed for brainstorming and debugging.

John Priscu (left) and Bart Hogan are bundled up for the drive home after work at 10pm.

The view out our back door - the Matterhorn peak on the north side of Taylor valley, taken from main Bonney camp.

Sunday
November 8, 2009

The snow storm went on through the night and into mid-day on November 8. Helo flights were grounded and areas around McMurdo reached Condition 1.


Given the very late finish the previous day most people slept until around 1pm. We left for the lab at 2pm to begin a detailed assessment of the situation that had caused the battery system to shut down at the end of yesterday's mission. A team meeting was held - always with everyone standing to get to the point - and tasks were assigned.


Vickie and Bill headed off on an ATV with the GPS equipment to locate all of the surface fixes obtained during the tracking of the vehicle. Bill then reduced these data to calculate circular error probable (CEP) for the re-navigation to the 2008 grid points (the main purpose of this year's mission is to duplicate last year's data set to observe temporal change in the lake chemistry). The CEP was 1.92 meters, meaning on average the bot navigated to within that radius of the exact GPS point reached last year for the sonde grid points. Some (perhaps the majority) of this error was related to the method by which sonde points were acquired: the vehicle navigated to the grid point coordinates where it then went into "station keeping" where the thrusters maintained that position horizontally while allowing it to rise until it stabilized against the underside of the ice cap (the vehicle was always ballasted about 400 grams positively buoyant for this purpose). This condition is referred to as "ice picking". At that point the thrusters would shut down. The vehicle is equipped with ice picking stand-off posts with rounded ends with very low friction resistance. So if there is any gradient to the underside of the ice (which is not uncommon) the bot can slide a meter or more on occasion before coming to rest. Even with this variation, the re-location navigation error was well within the five meter radius defined by the science team (Peter Doran and John Priscu) as being good enough for a match to the previous data set. So yesterday's data set was good, both from the chemistry information acquired as well as its geo-spatial position.


Meanwhile, Bart, Chris, Shilpa, and Kristof went through a detailed review of yesterday's onboard data log files to reconstruct the incident. The conclusion, as Bart had hypothesized yesterday, was that the batteries had automatically shut down just as the bot was stopping for a sonde point. The data logs showed clearly that the current draw on the power stacks reached 30 amps on fast stop - this depressed the single cell voltage of the lowest cell in the lithium stack and tripped the first battery safety latch. The second battery, similarly already reduced in power, was then tasked with all the remaining vehicle power load and it also then tripped - at an apparent 20% remaining power. Why there would be additional "stranded" power is a separate issue we will investigate in the coming days. In the meantime, Kristof began writing a current-limiting behavior for starting and stopping on high speed transit. Theoretically the voltage depression spiking will be reduced if we limit the thruster current draw and allow us to utilize closer to 100% of the battery. With the batteries now re-charged the system came up normally with all communications re-established and all sensors working normally. We are planning current-spiking tests following the conclusion of tomorrow's long sonde mission.

Blog entries from previous weeks can be found below:

November 1 - 7, 2009

October 6 - 31, 2009

Personnel:

Dr. Peter Doran is the Principal Investigator for the Endurance project. He has been studying the lakes in the Antarctic Dry Valleys for 15 years and is a professor at the University of Illinois, Chicago.

Dr. Bill Stone is a Co-Investigator for Endurance. He is also the president and CEO of Stone Aerospace, the engineering company that designed and built the Endurance vehicle.

Dr. John Priscu is a Co-Investigator from the University of Montana. He has been studying the lakes in the Antarctic Dry Valleys for over 20 years.

Bart Hogan is a mechanical engineer for Stone Aerospace.

Dr. Kristof Richmond is a programmer for Stone Aerospace, focusing on navigation.

Shilpa Gulati is a programmer for Stone Aerospace, focusing on the system executive and machine vision. She is also a PhD student at University of Texas, Austin.

Chris Flesher is the vehicle manager, programmer, and electronics technician for Stone Aerospace. He is a Masters student at UT Austin.

Rachel Middleton Price is an electronics technician for Stone Aerospace.

Vickie Siegel is chief vehicle technician and logistics manager for Stone Aerospace.

Maciek Obryk is a graduate student at the University of Illinois at Chicago.

Emma Steger is the Lake Bonney camp manager.