The 8 O’clock Report – Alpha Health & Status – 19 June 2001 Part 2

Samples of ITCS fluid were removed from the two operating ITCS loops during flights 5A, 5A.1, and 6A and returned to ground for analysis. The 5A samples showed no ammonia contamination. The 5A.1 samples had an ammonia concentration of 0.03 to 0.05 ppm (estimated due to this level being at the boundary of the detectable limit) in the low temperature (LTL), and the moderate temperature loop (MTL) was 0.09 ppm. The 6A samples have a 0.096 ppm, MTL; and a 0.142 ppm, LTL. The concern is that the trace amounts of ammonia contamination are being introduced to the fluid streams by a “micro-crack” or fissure within the single barrier interface heat exchanger (IHX). A “micro-leak” is not a catastrophic hazard to the crew. Threats to the hardware ( both internal and external) are unknown at this time. Once all possible sources of ammonia contamination are investigated, and if it is determined that the ammonia contamination is from an IHX, then the deficient heat exchanger will need to be removed and replaced by EVA operations. Presently, a spare IHX is stored in the USL. Procedures will have to be developed to remove and replace the IHX. Ammonia contamination in the ITCS fluid could come from: 1) sample bag material/contamination; 2) cabin atmosphere; 3) ground laboratory processing; 4) microbial activity; and/or 5) interface heat exchanger “micro-leak”. A team of engineers/scientists has been formed under the leadership of Cynthia Cross (JSC-EC) and Steve Daugherty (Boeing Active Thermal Control Team). The team consists of representatives from System Safety, Materials & Processes, and Engineering, with specialists in fracture control, chemistry, and fluid hydraulics. A detailed plan is being formulated to investigate potential sources of ammonia contamination. Samples will be taken and returned to ground for analysis at 7A and 7A.1 for trend analysis. It is noted that the quantities of ammonia contamination are “trace” amounts. The ITCS soft materials were tested at an 8.87% ammonia solution by weight ( Reference: Test Report Nonmetallic Material Compatibility with Ammonia , issued by Materials and processes laboratory, Marshall Space Flight Center, date: 10/96) for time periods of 30 and 60 days with all materials in contact with the ITCS fluid showing no significant degradation. 05/30/01: JSC is currently investigating (designing and funding) a test with ITCS fluid in a FEP sample bag to confirm the adequacy of the bag material in preventing ammonia from being absorbed from the cabin atmosphere. Investigation Continues.

On GMT 2001/122:07:30 the 4B BGA recorded sustained current greater than 1 amp for 4 minutes. This condition met the established guidelines for stopping rotation; therefore, the rate-mode Sun-tracking was terminated and the BGA was rotated to the optimal stationary power generation angle of 140 degrees. During this positioning the BGA experienced a high resistance to motion which caused excess motor current to be generated. The BGA was able to reach its intended location, however, it instantly overshot 2.5 degrees and began to slowly decay the high residual current. Due to the magnitude of the current, the BGA was unable to reduce this current and rebuild it in the opposite direction. Therefore, a motor stall trip was recieved as was expected for this situation. The 4B BGA is currently to remain fixed unless rotation is necessary for one of the following reasons:

1) Relocation to a improved stationary power generation angle

2) Feathering for Shuttle Dock/Undock

3) Feathering for Waste-Water Dumps

4) Feathering to prevent KURS antenna blockage during Russian Docking (If Necessary)

Keeping 4B in a fixed location while the station is in an XVV attitude may require power loads to be transferred from the 4B channel to the 2B. In peak power generation intervals it may be necessary to perform powerdowns to retain positive power margin.

PRACA PR# 2685 has been generated to specifically track this issue. 4B will stay in directed position mode. Team is investigating flying a replacement BMRRM on a future flight (7A.1).

The IV CPDS A-detectors have been having too many interrupts during operation. This may be due to the threshold being set too high. To properly evaluate this problem, data must be dumped to the ground and assessed. The ground successfully completed IV-CPDS data dump. Radiation will be evaluating data. If the thresholds are set too high, the only way to change them is through a software change. Update (5-16-01) Data dumped to the ground is being assessed. Preliminary assessment indicates that there are nearly 37% zero ADC values in A1 and A2. A higher threshold value should eliminate these events and some additional background events considerably and allow IV CPDS to continue functioning past the 24-hour mark. It was also noted that there was a very high counting rate in the Cherenkov detector.

The instrument has been commanded to standby mode, which will allow downlink of data for analysis.  Analysis of the data should help isolate the problem and allow corrective actions to be performed. Update (5-16-01): Data analysis indicates that the threshold settings are too high thus causing a large amount of interrupts. Investigation continues regarding the involvement of the very high counting rate in the Cherenkov detector.

Update (5-31-01): Preliminary analysis of the 240 data files downloaded so far indicate that the IV-CPDS A1 and A2 detector coincident rate is anomalously high, which generates a lot of interrupts.  With so many interrupts, when the elapsed timer hits 24 hours, the unit hangs rather than performing its normal 24-hour reset.

Update (6-1-01): BME tried unsuccessfully to downlink IV CPDS data files. It is uncertain at this time whether the problem was onboard or on the ground.

A CR will be submitted to modify the software to increase the gain settings so that the high level of noise will be eliminated.  In order to load this software on-orbit, the MEC software will also have to be changed to add an RS-232 data transfer program.  Until new software is loaded, the IV CPDS will be left in standby mode, available for activation in the event of a significant increase in radiation.  It is capable of short periods of monitoring if required.

Update (5-31-01): The only way to get the unit back to normal operation is to modify the threshold settings in the software. This cannot be accomplished without installing a new software load on the IV CPDS, most likely through the MEC via RS-232 communication.  Due to this fact, leaving the IV CPDS in the standby mode is the optimum configuration, as it will prevent it from getting hung at the 24 hour mark, and it remains available for periodic short duration monitoring of radiation, should the ISS environment and/or crew health concerns so dictate. 

In order to expedite a new software load on-orbit, several tasks must be completed quickly: 1) a software update must be developed on the ground; 2) the software must be tested in standalone mode and at the Software Verification Facility (SVF); 3) the software must be uplinked via OCA to the MEC; 4) the MEC software load/configuration must be updated to include an IV CPDS data transfer program; 5) the software update transfer to the MEC must be timelined. 

The only immediate activity that can be performed to help recover the IV CPDS is to downlink the remaining IV CPDS data files as soon as possible.  This will ensure the proper gain setting change for the IV CPDS software modification.  A software upgrade is already in work. Support from the Software Verification Facility is being requested.

Update (6-12-01), based on the data obtained to date, it appears that the IV CPDS is either not responding properly or not receiving the correct command.  This type of problem has never been seen on the ground through SVF testing or ORT testing with the ground unit.  The problem described for the IV CPDS has some similarities to the problem described for the TEPC, though the signatures are not identical.  The IV CPDS is experiencing/responding to one of 5 possible failure modes: 

1. IV CPDS software bug.

2. IV CPDS hardware failure.  Likely candidates include the 1553 board or the memory board.

3. IV CPDS Power/Data cable failures.

4. Payload bus problem between PL MDM and UOP.

5. Payload MDM output problem.

In order to better isolate the problem, troubleshooting commanding and crew actions are requested as described below.  The steps below are described in chronological order:

1. Power cycle IV CPDS and command to Standby mode as quickly as possible.

2. If step 1 successful, command IV CPDS to perform a BIT test on the 1553 card and the memory card.  Dump the BIT data from the MDM for ground analysis. 

3. Connect MEC to UOP 2-J3 (UOP TEPC was plugged in to) and configure for 1553 communication per MEC – 1553 Card/Cable Assembly, Steps 1-6 (7A MEDICAL OPERATIONS).  As part of procedure, verify MEC is rebooted or powered on after removal of the LAN card. 

4. Perform mode commanding  to MEC (Standby to Acquire to Standby) at least 5 times over 48 hour period.  Verify MEC cyclic data command response echo indicates correct command was received.

Crew reported a rhythmic scraping/grinding sound in the front end, possibly on the left hand side. The sound is only heard at low speeds (2-3 mph) during motorized exercise. It is not heard at higher speeds nor is it heard at any speed when used in passive mode. Update (5-14-01) Troubleshooting procedure sent to crew, which included requests for video, photos and PC card data recorded during operation of the TVIS without a runner. MCC downlinked PC card data. Flight Project currently studying the all information media, along with crew comments (including stethoscope observations) to determine cause of problem. The crew reported that the noise appears to be coming from a section of the TVIS 2″ inside the transfer case and 2″ down from the surface of TVIS. They said it sounds like a diesel motor inside a diesel-powered ship.

Update (5-16-01): The data from the TVIS PC data card was analyzed and it indicated that the crew did not operate the TVIS above 7.5 mph as requested in the procedure. A report from the crew this morning confirmed that the crew commanded the TVIS to 10 mph but the data is not indicating that the TVIS responded. Also a sound wave file of the most likely motor box problem was uplinked to the crew for them to confirm it is the sound they heard. The crew called down this morning and reported that the sound wav uplinked was exactly what they were hearing. Update (5-17-01) Engineering has isolated the problem to a failed output phase inside the motor box. Further isolation inside the motor box is in work. The corrective action is to replace the motor box ORU with the backup unit. Further work must be done to identify the root cause of the problem to ensure it does not happen again. Update (5-21-01) Continue using TVIS in passive (non-motor driven) mode. Fly replacement motor on 7A.1

The crew was trying to do a Physical Fitness Evaluation (PFE) in the USL but the UOP the BP/ECG was plugged into (UOP 2) tripped its circuit breaker.  With some questions, we found out that the circuit breaker tripped when they turned on the UOP with the BP/ECG powered off.  They switched to the backup BP/ECG power cable and repeated the same problem. 

The crew tried several different troubleshooting steps and found out that 1) when the MEC is disconnected from the UOP, it does the same thing and 2) when the spare BP/ECG power cable is used, it does the same thing.  We verified that the RS-232 cable was not connected between the MEC and BP/ECG when the breaker tripped, so that was not a contributing factor.  Finally, the crew ran a UOP malfunction procedure that validated the UOP was functioning properly without BP/ECG connected.  I should also add that TEPC has been plugged  into this same outlet for the past month without any problems.

In investigating the UOP, there are two possible explanations for why it might trip.  First, it would trip if the ground leakage current exceeded 8 mA.  Second, if it has a high startup current spike, it would trip.  Since BP/ECG was powered off, and since it has a fuse in the cable itself, it is highly unlikely that it experienced a high startup current spike.  The more likely explanation is a ground leakage current problem, either from BP/ECG or the UOP limit being lower than expected.  Based on the BP/ECG ground certification data, it should not have leakage current over 0.1 mA unless it had some sort of unusual hardware failure.

EXTRAVEHICULAR ACTIVITIES GROUP (EVG)

No open IFI’s to Report

After a series of video routing commands, the SSRMS Tip Elbow camera became routed to both channel 1 and channel 2. Then the SSRMS Base Elbow camera was routed to channel 1 also. This left 2 MSS cameras routed to the same channel. When all views were subsequently de-routed, the SSRMS Tip Elbow camera was left connected to channel 1.

A subsequent command to route outboard SSRMS cameras (Tip LEE) on channel 1 would not result in a video picture from that camera until the Tip Elbow TVC is de-routed from channel 1.

Event 13/16:25:00 Base Elbow camera routed to MSS Video line 1 in spite of Tip elbow camera already routed to that line.

Routings up to the event point appear normal until the following:

113/13:45:32 OCS> De-route Video Accepted

113/12:45:35 OCS> Route Video Accepted

113/13:45:43 SSRMS_Tip_Elbow_VDU_PFM_Source_Select_Video_1 – TVC

113/13:45:43 SSRMS_Base_Elbow_VDU_PFM_Source_Select_Video_1 – Upstream

Note: that the Tip Elbow TVC was already routed on MSS video 2 (since about 113/10:40) so the Tip Elbow should not have routed to MSS Video 1 (the VSUs should simply have routed the MSS Video 2 line to the 2nd destination) Proposed solution is to ask ISS to establish sequence of commanding events leading to this state and to determine whether or not the C&C MDM commanded a De-route for the Tip Elbow camera from Video channel 1 at 16:25 when the Base Elbow camera was routed.