NASA LCROSS Flight Director’s Blog: Multi-Tasking: Day 4
Posted on Jun 26, 2009 03:52:36 AM Paul D. Tompkins
First, a note from the Flight Director…my apologies for not keeping up with my writing. I’ve been extremely busy with flight duties, and am just now getting on top of things again. This post was written in draft form a couple of days ago, but just now shaped up for posting. I am enthused by all of your comments and encouragement! Thank you for all the great notes. I hope you enjoy this next installment. More to come soon!
-Paul
The first week of the LCROSS mission plan is ambitious, even without problems to work through. On Day 3 when we started realizing we had some thermal issues, it was a little daunting. But now that I think about it, our team had met similar challenges prior to launch. Over the LCROSS development, our team has had to work around a lot of surprises, under a very tight schedule and budget. So, in a sense, development was training us for flight in ways we could not have anticipated.
At NASA Ames, after Day 3, Shift A (my shift) transitioned from “execution” to “planning”. As part of our staffing plan, Shift B exchanged roles with us to take TCM 3, Star Field Cal (Day 4) and Lunar Swingby (Day 5) execution. The execution shifts end up being fully committed to commanding maneuvers and other spacecraft activities, so when we started having our cold thruster issue, our Shift A became the de facto shift for coming up with solutions to the problem. You see, LCROSS was designed to manage itself thermally if we maintained its orientation correctly relative to the sun. The trouble is that our thrusters are getting too cold, even when we’re sun-pointed.
Operationally, this means until we design a good thermal control mode, we have a high-maintenance spacecraft. Our goal was to find a control approach that would enable our spacecraft to manage itself again. At first we had to change the orientation of the spacecraft every 2-3 hours, sometimes more. Mixed in with our other numerous activities, this was really difficult to manage. In collaborations between NASA Ames and our partners at Northrop Grumman, we started experimenting with different orientations, warming one side, warming the other side, entering control deadbands to fire thrusters more often, etc. No strategy worked for more than a few hours. Then we found an orientation that was more effective than the rest – where we tipped the “top” of our spacecraft a little bit toward the sun (negative yaw angle). The thrusters at the bottom of the spacecraft were warmed by heaters, and the ones at the top, whose heaters don’t activate very easily, were warmed by the sun. On my Day 3 post, we tipped 10 degrees to the sun (-10 deg in yaw), and things looked great. I came back on Day 4, and found to my disappointment that the thruster temperatures fell too low after several hours. But, this approach was a great discovery – maybe if we pushed it a little further, things would balance.
For Day 4, we had to improve our thermal performance, and also oversee the planners in creating commands for TCM 3 and Star Field Calibration. We set to work. Our first experiment was to nudge our attitude a little further in yaw, to -20 degrees. Technically, this violated a flight rule that protects our spacecraft against thermal damage. The LCROSS spacecraft was designed to fly with its solar array pointed directly towards the sun. Outside the bounds set by our thermal flight rules, the concern was that perhaps the parts of the spacecraft newly exposed to the sun would overheat, and those newly shadowed from the sun would get too cold. But confronted with an urgent problem, the flight team had to explore options that were outside of the nominal flight envelope. Our hope was that the thermal subsystem was designed with some conservatism, and that it would allow us to off-point from the sun just a little bit.
So, we rotated to -20 degrees yaw, and watched, and waited. The behavior we observed at -10 degrees on Day 3 improved, and magically, the bad effects we feared might bite us never emerged. In the short term, the -20 degree yaw bias seemed to do the trick. Thruster temperatures on the top of the spacecraft really stabilized, and those on the bottom continued to be warmed by their heaters. From there, our engineering team wanted to explore the thermal sensitivity to these yaw changes, to understand how vulnerable we’d be if, for whatever reason we moved even further in yaw away from directly sun-pointed. Would we get too hot, or too cold? We needed to know, so we commanded the spacecraft just a bit further, to -25 degrees.
Unfortunately we were so focused on thermal issues that we neglected to check the safety of this new attitude from other perspectives. As soon as LCROSS began rotating, our Star Tracker (STA) “faulted” and dropped into its Standby Mode. I knew immediately what we had done. As we pointed our solar array down, the STA, mounted on the opposite site of the spacecraft, pointed more directly to the Earth. The Earth is bright, and the STA cannot process star fields to determine our orientation if the sun, Earth or moon enter into its field-of-view. So, our spacecraft reverted to relying only on its inertial reference unit (IRU) for forward-propagating our attitude. No immediate threat. Our IRU is quite capable of keeping track of the spacecraft orientation, so none of us were immediately worried. However, I started worrying about TCM 3. If, for whatever reason, we couldn’t get the STA re-initialized soon, I wanted to know whether it would suffice for accurately achieving the burn orientation for TCM 3. As Flight Director, you always have to think ahead to the consequences of your actions. I was disappointed that we hadn’t anticipated the entry of the Earth into the STA field-of-view, but we needed to proceed calmly and to right the wrong.
Our first action was to return to -15 degrees, to make sure the Earth would not interfere with the STA and to begin the STA recovery process (blinding is temporary, but requires a few minutes). Of course by this time TCM 3 planning was going on, and the Flight Director on “planning” shifts has to help oversee the creation of new command sequences. I missed the initial coordination meeting that nails down the precise set of activities we’d be performing later that night, but caught up with the process soon afterward. Once I saw things were on track, I returned to the Mission Operations Control Room (MOCR) to continue working the STA issue.
As it turns out, the STA recovery process is a little more involved from our standard Cruise configuration than from our initial Sun Point Mode from Day 1. In Sun Point Mode, the process to initialize the STA takes only a few minutes. In Stellar Inertial Mode, one has to re-initialize the spacecraft’s attitude estimator, a piece of software that merges the data from the IRU and STA into a single, optimal estimate of our current orientation. Not a big deal, really, but time was beginning to get short. As it turned out, with other spacecraft activities we’d had to perform, the team did not have time to re-initialize the STA before the Command Approval Meeting (CAM) for TCM 3 and Star Field Cal. This meeting requires the presence of both outgoing and incoming teams, along with the entire planning and analysis teams, and of course our Science and Payload teams. The meeting is important, since it is the venue where everyone gets their last view of the command sequences before execution, to make sure all is in order. We suspended active spacecraft operations, and with a skeleton crew monitoring spacecraft health, we held the CAM. In the CAM, armed with strong evidence that the burn could be safely and accurately conducted without the STA, we resolved that if time for STA recovery ran out prior to the burn, we’d continue with TCM 3 anyway and recover it later.
With the CAM completed, time was critical. First, Shift A and the oncoming Shift B held a Shift Handover meeting to communicate all that had happened while B was gone, and to make sure Shift B was fully aware of the current state of LCROSS. With little time to spare, Shift B took over operations in the MOCR, working towards getting the TCM 3 commands loaded in time. I watched for a while, as Rusty, the Shift B Flight Director, moved to get everything done. One concern was that our DSN site had had equipment problems on the Shift B from the night before, preventing commanding for a significant period. Now we were over the same site, and we couldn’t afford an outage. Things were going well and then bam! Suddenly, there was a commanding outage, and things ground to a halt. Never a dull moment! Watching from the sidelines, I knew Rusty was in a real bind. The station worked to resolve the issue with its ground equipment, and all of us in the MOCR started sweating. But Shift B kept from panicking, and within minutes worked with DSN to get the problem resolved. They really pulled TCM 3 back from the brink. With only 10 minutes to spare (now too little time to recover the STA), Shift B got the commands into the spacecraft and the ball was rolling again! I congratulated the team, and headed off to bed, confident that TCM 3 would go off without a hitch.
It was a very satisfying moment, when the team really came together in a pinch to keep our operations on track. Great job Shift B!
