Status Report

The 8 O’clock Report – Alpha Health & Status – 19 June 2001 Part 2

By SpaceRef Editor
June 19, 2001
Filed under , ,

The 8 O’clock Report

Alpha Health & Status

June 19, 2001 (GMT 170/00:00)

(The information highlighted in blue is update to yesterday’s report)



  • ECLSS – (MER-0316)- Elektron Unexpected Shutdown.

The Russians reported that the problem was the Elektron power supply (power cable). They reported that they would command reactivation during their next ground pass; they expected this to be successful; if not successful, there is a spare power cable onboard. At GMT 2001/088:06:41:16, the Elektron Fail indication was reset. During an LOS between GMT 2001/088:13:06 and GMT 2001/088:13:22, the Elektron was deactivated (the Progress vehicle was available for Oxygen supply). Elektron has been returned to service and has been operating satisfactorily for several weeks. Part number and serial number requested from MER Moscow/Sean Melody on GMT 2001/150. PRACA PR# 2710 was generated to document this unexplained anomaly.

  • ECLSS – (MER-0319)- High Cadmium Levels in Water

Water samples taken from the SVO-ZV crew ambient water dispenser on Flight 4A exceeded the SMAC levels for cadmium. Samples taken on 5A.1 were lower than the 4A samples but exceeded SMAC levels. Potable water source is contaminated with cadmium. Source of cadmium was determined by the Russians to be the dispenser that flows from the SVO-ZV to the crews drinking water. The dispenser contained cadmium, which was released into the water as it was dispensed. Russians have plans in place to remove the dispenser and use the SRVK sample adapter to hook up to the SVO-ZV to dispense water. This sample adapter has been used for previous sampling and has shown to not contaminate the samples with cadmium. Source of contamination has been found and an on-orbit workaround has been developed to prevent continual contamination. It is recommended that a PRACA (need P/N and S/N) be opened for this problem, and the IFI can be closed to the PRACA when repairs have been made and another water sample proves that the cadmium source has been removed. PRACA PR# 2711 was generated to specifically document this issue. Subsequently, a MAR (ECLSS-00016-BASIC) has been issued to manifest the repair parts.

  • ECLSS – (MER-0328)- SM BB2PO Fan Failure

Crew reported the BB2PO fan failure at GMT 2001/94:20:29. The crew removed and inspected the fan. One of the fan blades was damaged. A 2.5 cm piece of debris was found in the vicinity of the fan when inspected. The debris is the most probable cause of the failure. Significance to vehicle: Loss of fan until replaced with another MO-1-5006 fan. Significance to ops: Crew is currently taking the BTK2 fan in the SM and installing it in the place of the failed BB2PO. A 2.5 cm piece of debris was found in the vicinity of the fan when inspected. The debris is the most probable cause of the failure. Fan has been replaced and is operating satisfactorily. To completely restore normal operations, a replacement fan for the BTK2 fan in the CKB2 side of the ventilation system. It is recommended that a PRACA be opened for this problem. P/N and S/N requested from MER Moscow on GMT 2001/150.

  • ECLSS – (MER-0339)- MCA Error Code 33 with INT Stop Cmd

John Carr (ECLS) reported the following: Error Code 33 occurred again. MCA went to RESTART. RESTART did not work. For some reason currently not understood, the INT sent a STOP command to the MCA and the MCA went to IDLE. MOD sent a STARTUP NORMAL command and the MCA went to OPEARATE with RAPID SAMPLING of the Lab (MOD did not want the MCA to lose its vacuum.). While in OPERATE, the MCA Location Accessible parameter went from YES to NO. At that time MOD commanded the MCA to STANDBY. The incident appears similar to the incident of last Saturday (GMT day 2001/104) except for the STOP command from the INT and the Location Accessible indicating NO. Problem caused by either:

  1. Software thinking it has a problem with the filament (caused by EMI, South Atlantic Anomaly or solar flares).

(2) The buildup of oxidized metal in the ion pump; periodically flakes of this oxidized metal will break off and float into the ion beam and cause a reading that exceeds the limit and causes the MCA to shutdown. Proposed solution – for cause (1) above, the solution would be to change the ion pump current limit. For cause (2) above, the solution would be a design change of a circuit card and a change out of the ORU. PRACA PR# 2682 has been generated. S/N#F0002.

  • ECLSS – (MER-0348)- VRS Leak rate higher than expected

During 5A.1 with VRS Vent Valve open, VRS System Pressure was approximately 250 microtorr. This is significantly higher than was observed on 5A. After closing VRS Vent Valve, VRS pressure rise rate was observed to be approximately 218 millitorr/hour. (This compares to a pressure rise rate of approximately 1.86 millitorr/hour after VRS Vent Valve closure during 5A). The “spec” leakage rate for the VRS is 4 x 10exp-4 scc/second. With the spec leakage rate, the VRS System Pressure would rise at a rate of approximately 18.6 millitorr/hour. Pressure rise rate data suggests the VRS was leak-tight during 5A, but may now be leaking. Pressure rise rate of 218 millitorr/hour is roughly consistent with a leakage rate, which would produce 250-microtorr pressures at the VRS CCT with the Vent Valve open. When crew time available (e.g., during 6A stage) disconnect HRF Payload from VRS and cap the VRS QD at this location. Observe whether this eliminates the leak. There is no urgency to do this. There are no payloads, which need to use the VRS through at least 8A. (Although we should do this as soon as conveniently feasible, since if HRF is not the problem, we will have more time to resolve the problem).

  • ECLSS – (MER-0427)- Node 1 Smoke Detector Obscuration and Scatter High

Node 1 Smoke Detector (SD) Obscuration & Scatter High. At activation after a lengthy shutdown, Node 1 smoke detector #2 had Scatter reading of 0.55 and obscuration reading of 8.55. Prior to the SD being shutdown, scatter and obscuration read 0.83 and 5.82, respectively.

Node 1 suspect. Not enabled to produce fire caution/warning. Operation with SD 1 only is acceptable. As of GMT 2001/169:16:30:00, the SD remained powered, but inhibited from producing alarms. SD 2 ORU may have to be replaced. Under Investigation.

  • ECLSS – (MER-0428)- MCA Purge Time Override Failure and Loss of Synchronization with the INT MDM

On GMT 163, the MCA INT status went to from Operational to Purge time override failed while sampling the Node. The Waiting Confirmation status changed to waiting and was frozen in that configuration. Due to this fact, the MCA would not accept commands to restore it to an operational status. The MCA was later power cycled to clear the waiting status. By this time, the MCA mass spec vacuum level had increased to the point where a pump-out was required. The MCA has an error code 24 for mass spec OOL. A temporary loss of U.S. Segment capability to monitor O2, N2, CO2, H2, and CH4 has occurred will remain until the MCA ORU R&R is complete and the MCA is reactivated. This same anomaly also occurred on GMT 118 following the C&C crashes.


  • ECLSS – (MER-0429)- Leaking CWCs

Crew reported five CWCs with fungus growth. Fungus cleaned with fungicide and CWCs moved to Node 1 for observation to determine which one(s) is(are) leaking. Crew later reported only one CWC appeared to be leaking and it’s serial number is 5093. It was also reported it was wet at the seam next to one of the two valves (the valve with black rubber around it). Outer cover of leaky CWC and nearby CWCs are wet and have mildew growth. Leaky CWC may have to be drained due to a hole in the CWC inner liner or connecting tubes. Under Investigation.



  • C&T — (MER-0259) – Echo on S Band (Ground).

When CAPCOM is talking to the crew in the SM, the ground hears an echo. The ground hears a repeat of their voice. The ISS audio system was configured with S/G 1, ATU Lab 1, ATU Lab2, and RSA 1 in Public Loop1 and S/G 2 ATU Lab 1, ATU Lab 2, and RSA 2 in Public 2. It is probable that the SM audio system was configured in a way that the voice was looped back down the S/G channel.

Forward plan for closure:

OCA_1578 US/RS Comm Configuration procedure was uplinked to the crew to give them an integrated audio config procedure and integrated audio diagram to help them understand the system better. CATO believes that the audio echo is a product of how the RSA1 and 2 lines are connected to SM comm panel 3 and 2 respectively. CATO also wrote OCA 1652 — S-band Troubleshooting. The crew due to a full timeline has not performed it. It will be performed post 5A.1. C&T wrote 7A Chit ISS0089 to perform on board troubleshooting. This IFI will be closed based on results of this chit. Investigation continues.

  • C&T — (MER-0329)- VBSP Channel #4 Configuration did not go to Default

The VBSP went to self test when commanded to equipment self test, and the equipment self test status was active, Self test was terminated by issuing a deactivate equipment self test command. Upon termination equipment self test, the VBSP did not go to default configuration because it did not start in its default configuration (when the correct self test command is sent to the VBSP, it is expected to return to its default configuration). Currently, the command sent to initiate the VBSP self test command is INCORRECT (see PR15876). In all of our testing, we always ran the self test right after the box was initially powered up – i.e. started in the default configuration, and when it was sent the deactivate equipment self test command, it just returned to where it started. We suspect that’s all it’s doing now returning to the state it started in.

We believe this IFI should be closed because the software will be fixed at Flt 8A with the CCSR2 load. We could probably do some investigative testing at ISIL with the DVTM but we don’t believe that’s really worthwhile because we have a debugging plan referenced above, and the s/w will be fixed at Flt 8A and this won’t be an issue anymore.

  • C&T — (MER-0333)- MCOR (Medium Rate Comm. Outage Recorder) Upper Temperature Accidence.

When MCOR Activation & Check-out procedure was being performed; on step 7, MCOR had to shutdown due to over temp. MCOR was activated at approx. 101:23:02GMT (6:23 pm CST??). Two minutes later we received heartbeat indication.

Steps 1-6 were completed from the procedures flawlessly. Commands accepted, RT enabled, verification of default configuration and health, documented memory configuration, initiation of BIT test and verification of BIT results. At the step 7 (enabling recording of input channels 4 and 8), the overtemp indication flag came up and MCOR had to shutdown. It happened approximately 45 minutes after activation. Investigation continues. ART will come up with the rational to close this IFI ECD 6/08/2001.

  • C&T – (MER-0355) — Broken ECOMM Connector

The Early Communication System (EComm or ECS) Starboard Antenna was removed during the 6A EVA2. When the EComm data and power cables were disconnected, part or parts came loose from the cable connectors and floated away. One or more part(s) moved towards the CBM. [IFI 0352 was written to investigate, track, and resolve the Foreign Object or Debris (FOD) situation.] (The EComm Starboard Antenna Assembly P/N is SEG39130674; the Power/Heater cable is P/N SED39129744-301, and the Data/OCA cable is P/N SED39129747-301.) Apparently connector pins and/or screw/bolts were dislodged allowing the connector(s) to emit part(s). The proposed solution is to obtain the culprit cables (power and data) of the returned EComm Starboard Antenna and provide an evaluation and analysis to determine the condition of the cables and connectors, and the likely cause(s) for the connector part(s) coming loose. This IFI can be closed because EComm is GFE, a PRACA should be written to track this issue. The problem is likely a design, manufacturing, or readiness for flight issue. Since the EComm is has been de-commisioned on 6A, the connector design or manufacturing issues are moot. No future use of the EComm is planned for ISS/Station. Recommend that this IFI be closed via FIAR # JSCEV1270F. Awaiting return of data and power cables to determine parts missing and to assess cause. This IFI is CLOSED as of 06-15-01.

  • C&T – (MER-0400) — Regul Acquisition of Signal Error

Working with Russians / MER Moscow. This IFI Under Investigation.

  • C&T — (MER-0421) – Ku Band PLC to Normal in Mask

As reported by CATO, a “Ku-Band Radiating Beyond Mask” warning was annunciated at GMT 2001/156:18:27:30. As expected the Ku-Band TRC was powered OFF. CATO RTPLOT indicates that TRC PLC went to “Normal” at 156:18:27:31. The Ku-Band PWRL was at -57 dBm when the PLC went to “Normal”. The TRC Xmit Output voltage was at 0 volts (therefore, we did not radiate within the protected Mask. CATO performed CAFN576, Ku-Band Radiating Beyond Mask Corrective Actions, in order to recover the Ku-Band TRC. Investigation continues.

  • C&T — (MER-0425) – Ku-band PLC Reset

A TD171 to TD171 Single Access handover occurred at GMT 2001/163:09:26:47. Prior to the handover, the Ku-Band system was operating nominally in autotrack mode. After the handover back to TD171 at 163:09:27:07, a TRC Execute function Configuration command was sent at 163:09:28:06 to configure the Ku-Band back to autotrack mode. The command was accepted and the system transitioned to open loop slew then began the spiral acquisition sequence as expected. The TD171 fwd link signal was found during the spiral search and the system transitioned to acquisition lock state with a PWRL of —47 dB, The PLC state transitioned from reset to normal momentarily, but then transitioned back to reset. As a result, the Ku-Band transmitter output power remained at zero. The Ku-Band Antenna was neither near a mask nor did the PWRL value drop below —51 dB at any time after the initial acquisition sequence. The system remained in autotrack mode and continued to track the TD171 FWD link until the end of the TDRS event at 163:09:33:57. The Ku-Band PWRL level remained at a level of —46dB and PLC state remained in reset for the entire event. When the TDRS handover began the TDRS signal strength dropped below the CCS monitored threshold so the C&C MDM commanded the antenna to Pointing Mode Inhibit and PLC to Reset as part of the “Drop Lock” CCS patch. When the handover was complete the CATOs commanded the antenna back to Autotrack, and the antenna successfully locked to the TDRS signal. When the antenna declared lock to the TDRS signal the TRC firmware put the PLC in Normal. At this same time the C&C MDM commanded the antenna back to Pointing Mode Autotrack as part of the CCS “Drop Lock” patch. As a side effect of this command the PLC was taken back to Reset.

Any time the Ku-band losses the TDRS signal, the C&C MDM is going to declare “loss of lock”. Two minutes later, regardless of the state of the antenna, the C&C MDM is going to command the Pointing Mode back to Autotrack. The last CCS commanded state of the PLC will also be included in this command. Every time the Ku-band antenna losses the TDRS signal, the C&C MDM will command it back into Autotrack after two minutes. This command will also include the last state of the PLC which should be Reset when we are in Autotrack. After short (<1 minute) TDRS handovers let the CCS software command the Ku-band antenna back into Autotrack. The CCS patch works as designed.

This IFI is closed as an explained condition as of 06-16-01.

  • C&T — (MER-0434) – Spare SASA RT Conflict

The prime SASA and spare SASA share a common 1553 RT address. They also share a common 1553 bus. Heater power is applied to the spare SASA to ensure that its temperature is maintained above the non-operating minimum level, but operational power is not applied to the spare SASA. If operational power were applied to the SASA, loss of ISS S-band communications would result. This is because both the spare and prime SASAs would attempt to communicate with the controlling MDM over the same bus from the same RT address at the same time. If operational power is applied to the spare SASA, ISS S-band communication will be disrupted. This will result in loss of station commanding and audio. A flight rule that reads “Do not close the RPC that delivers operational power to the spare SASA” will be written.

The future installation of the Early Ammonia Servicer requires the installation of a pair of “Y” cables to the connectors that provide heater power and operational power to the spare SASA. A procedure will be forthcoming explaining how to connect these Y cables to preclude the possibility of inadvertently applying power to the spare SASA. Under Investigation.

  • C&T — (MER-0435) – VTR-1 Operational Problem

An attempt was made to record and playback on VTR-1 but symptoms remained. Also, VTR-1 cannot playback a tape which VTR-2 can successfully playback. Currently, we think that VTR-1 record and playback capability is lost. A procedure is being considered to swap VTR-2 into VTR-1’s location to confirm that the CVIU for VTR-1 is working correctly. Since successful video can pass through VTR-1 to the downlink, it may not be necessary to perform a VTR swap/test. VTR-1 may have to be replaced during a future assembly mission.

  • C&T — (MER-0436) – Prime and Spare SASA (S-Band Antenna Structural Assy) RT Address Conflict

Under Investigation

  • C&T — (MER-0437) – SSRMS Dynamic Grapple Fixture Release

During un-grapple and back away from Lab FRGF, SSRMS was observed to jump considerably. Early indications are that this may have been caused by stored energy in the SSRMS (as reflected by motor currents well in excess of 0.2 amp guideline) and that this stored energy may have resulted from thermal environment changes (came out of XPOP during time grappled). Hand controller cross-coupling may have been a contributing factor. A much smaller jump was observed at 167/16:40 while WY current was around 0.4 amps. Limping is requested just prior to any un-grapple (CHIT) until situation is better understood.

Under investigation.



  • TCS — (MER-0403) – Ammonia Contamination of Lab ITCS Fluid

Samples of ITCS fluid were removed from the two operating ITCS loops during flights 5A, 5A.1, and 6A and returned to ground for analysis. The 5A samples showed no ammonia contamination. The 5A.1 samples had an ammonia concentration of 0.03 to 0.05 ppm (estimated due to this level being at the boundary of the detectable limit) in the low temperature (LTL), and the moderate temperature loop (MTL) was 0.09 ppm. The 6A samples have a 0.096 ppm, MTL; and a 0.142 ppm, LTL. The concern is that the trace amounts of ammonia contamination are being introduced to the fluid streams by a “micro-crack” or fissure within the single barrier interface heat exchanger (IHX). A “micro-leak” is not a catastrophic hazard to the crew. Threats to the hardware ( both internal and external) are unknown at this time. Once all possible sources of ammonia contamination are investigated, and if it is determined that the ammonia contamination is from an IHX, then the deficient heat exchanger will need to be removed and replaced by EVA operations. Presently, a spare IHX is stored in the USL. Procedures will have to be developed to remove and replace the IHX. Ammonia contamination in the ITCS fluid could come from: 1) sample bag material/contamination; 2) cabin atmosphere; 3) ground laboratory processing; 4) microbial activity; and/or 5) interface heat exchanger “micro-leak”. A team of engineers/scientists has been formed under the leadership of Cynthia Cross (JSC-EC) and Steve Daugherty (Boeing Active Thermal Control Team). The team consists of representatives from System Safety, Materials & Processes, and Engineering, with specialists in fracture control, chemistry, and fluid hydraulics. A detailed plan is being formulated to investigate potential sources of ammonia contamination. Samples will be taken and returned to ground for analysis at 7A and 7A.1 for trend analysis. It is noted that the quantities of ammonia contamination are “trace” amounts. The ITCS soft materials were tested at an 8.87% ammonia solution by weight ( Reference: Test Report Nonmetallic Material Compatibility with Ammonia , issued by Materials and processes laboratory, Marshall Space Flight Center, date: 10/96) for time periods of 30 and 60 days with all materials in contact with the ITCS fluid showing no significant degradation. 05/30/01: JSC is currently investigating (designing and funding) a test with ITCS fluid in a FEP sample bag to confirm the adequacy of the bag material in preventing ammonia from being absorbed from the cabin atmosphere. Investigation Continues.

  • TCS — (MER-0411) — ESP Survival Heaters Off Since 6A

Power was removed from the ESP-1 secondary heaters at 2001/116:15:51 and they were left off until it was discovered on day 147. ESP-1 houses a spare PFCS and a spare DSCU. During flight 6A, NCS control was lost and RACU 6 was power cycled at 2001/116:15:51 to remove and reapply power to the N1-1 MDM in an attempt to regain NCS control. The RACU 6 power cycle caused power bus N1RS1 and all its loads to lose power. The 6A N1RS1 Power Bus Repower procedure was missing the ESP heater load on RPCM N1RS1-B RPC 4. Consequently, the ESP-1 secondary heaters were left off until it was discovered on day 147. Also during flight 6A, due to bad NCS data, the DDCU Z1-3B had an NCS software initiated trip at 2001/117:02:13. This caused power to be removed from power buses N13B and N1RS2 and all their loads. The 6A N13B Power Bus Repower procedure was missing the ESP heater load on RPCM N13B-A RPC 3. Consequently, the ESP primary heaters were left off until it was discovered on day 147. ART provided thermal analysis indicating PFCS violated acceptance and qual test levels ( lowest PFCS calculated temp was -72 F vs accep/qual limits of -45 F / -65 F ). Lowest calculated DCSU temp was -57 F vs. accep/qual limits of -45 F / -65 F. Issue transitioned to the SPRT for recommendation regarding R&R of these components.

  • TCS — (MER-0418) – EETCS FCV Transient Transition to By-pass

On GMT 2001/152:08:54 and approximately 6 seconds after the Primary Node 1 MDM detected a loss of sync with the Primary PVCU MDM, the EETCS Loop A Flow Control Valve (FCV) went to the full bypass position and stayed there for approximately 10 seconds and then returned to its nominal expected position in closed loop control. The EETCS Loop B FCV carried out identically the same actions, but did so less than 1 second after the Primary Node 1 MDM detected a loss of sync with the Primary PVCU MDM. This was a one time transient event that did not affect the nominal operations of the hardware. There are currently no identified impacts to safety, hardware/software or operations. Investigation continues. Actions assigned to CP ( Kevin Graves ) to analyze and identify cause. Telemetry data and event description have been supplied to CP GMT159. ECD for analysis is GMT 173 ( 2 weeks ). Under investigation.

  • TCS — (MER-0431) – ESP Primary heaters failed to come on.

On GMT 2001/147:12:58 RPCM N1RS1-B RPC4 which controls the ESP-1 secondary heaters was closed. The amperage of RPCM N1RS1-B showed an increase that indicated 3 of 4 ESP-1 secondary heater elements came on. There are 2 secondary and 2 primary heater elements for both the PFCS and DCSU ORU’s. Predicted temperature of the PFCS and DCSU indicate that all of the secondary heater element should have come on when the RPC4 was closed. ART to meet 6/18/01 1 PM. Under Investigation.



  • S&M – (MER-0367)- Lab Window Contamination

Lab window cover was left open during Soyuz docking. Subsequent inspection suggested possible contamination. Further inspection was requested. The window cover had been requested to be closed for docking events per an existing procedure. Contamination poses a direct threat to the optical life of the outer window and a secondary threat to possible contamination when changed out. The problem was caused by combustion product contaminants from thruster firings. Procedure is in place to establish rules for closure in order to minimize accumulated contaminants for all flights. Special rule is drafted for flight 7A. Problem is understood. Contamination analysis is under way, assessment is complete. Rule is in place for 7A, general rule to be drafted for all flights. Recommend closure to PRACA. Problem is understood. Contamination analysis is under way, assessment is complete. Rule is in place for 7A, general rule to be drafted for all flights. PRACA PR# 2709 was generated to specifically address this issue.

  • S&M – (MER-0371)- BGA 4B High Torque Indication

On GMT 2001/122:07:30 the 4B BGA recorded sustained current greater than 1 amp for 4 minutes. This condition met the established guidelines for stopping rotation; therefore, the rate-mode Sun-tracking was terminated and the BGA was rotated to the optimal stationary power generation angle of 140 degrees. During this positioning the BGA experienced a high resistance to motion which caused excess motor current to be generated. The BGA was able to reach its intended location, however, it instantly overshot 2.5 degrees and began to slowly decay the high residual current. Due to the magnitude of the current, the BGA was unable to reduce this current and rebuild it in the opposite direction. Therefore, a motor stall trip was recieved as was expected for this situation. The 4B BGA is currently to remain fixed unless rotation is necessary for one of the following reasons:

1) Relocation to a improved stationary power generation angle

2) Feathering for Shuttle Dock/Undock

3) Feathering for Waste-Water Dumps

4) Feathering to prevent KURS antenna blockage during Russian Docking (If Necessary)

Keeping 4B in a fixed location while the station is in an XVV attitude may require power loads to be transferred from the 4B channel to the 2B. In peak power generation intervals it may be necessary to perform powerdowns to retain positive power margin.

PRACA PR# 2685 has been generated to specifically track this issue. 4B will stay in directed position mode. Team is investigating flying a replacement BMRRM on a future flight (7A.1).


  • S&M — (MER-0389) — BGA 2B Current Spikes

PRACA PR# 2685 has been generated to specifically track this issue.

  • S&M — (MER-0410) — IWIS Data Collection Anomaly

Under Investigation



No Open IFIs to report



No open IFI’s to report



  • CREW EQUIP – (MER-0376)- CHeCS IV-CPDS Display Data Errors

The IV CPDS A-detectors have been having too many interrupts during operation. This may be due to the threshold being set too high. To properly evaluate this problem, data must be dumped to the ground and assessed. The ground successfully completed IV-CPDS data dump. Radiation will be evaluating data. If the thresholds are set too high, the only way to change them is through a software change. Update (5-16-01) Data dumped to the ground is being assessed. Preliminary assessment indicates that there are nearly 37% zero ADC values in A1 and A2. A higher threshold value should eliminate these events and some additional background events considerably and allow IV CPDS to continue functioning past the 24-hour mark. It was also noted that there was a very high counting rate in the Cherenkov detector.

The instrument has been commanded to standby mode, which will allow downlink of data for analysis.  Analysis of the data should help isolate the problem and allow corrective actions to be performed. Update (5-16-01): Data analysis indicates that the threshold settings are too high thus causing a large amount of interrupts. Investigation continues regarding the involvement of the very high counting rate in the Cherenkov detector.

Update (5-31-01): Preliminary analysis of the 240 data files downloaded so far indicate that the IV-CPDS A1 and A2 detector coincident rate is anomalously high, which generates a lot of interrupts.  With so many interrupts, when the elapsed timer hits 24 hours, the unit hangs rather than performing its normal 24-hour reset.

Update (6-1-01): BME tried unsuccessfully to downlink IV CPDS data files. It is uncertain at this time whether the problem was onboard or on the ground.

A CR will be submitted to modify the software to increase the gain settings so that the high level of noise will be eliminated.  In order to load this software on-orbit, the MEC software will also have to be changed to add an RS-232 data transfer program.  Until new software is loaded, the IV CPDS will be left in standby mode, available for activation in the event of a significant increase in radiation.  It is capable of short periods of monitoring if required.

Update (5-31-01): The only way to get the unit back to normal operation is to modify the threshold settings in the software. This cannot be accomplished without installing a new software load on the IV CPDS, most likely through the MEC via RS-232 communication.  Due to this fact, leaving the IV CPDS in the standby mode is the optimum configuration, as it will prevent it from getting hung at the 24 hour mark, and it remains available for periodic short duration monitoring of radiation, should the ISS environment and/or crew health concerns so dictate. 

In order to expedite a new software load on-orbit, several tasks must be completed quickly: 1) a software update must be developed on the ground; 2) the software must be tested in standalone mode and at the Software Verification Facility (SVF); 3) the software must be uplinked via OCA to the MEC; 4) the MEC software load/configuration must be updated to include an IV CPDS data transfer program; 5) the software update transfer to the MEC must be timelined. 

The only immediate activity that can be performed to help recover the IV CPDS is to downlink the remaining IV CPDS data files as soon as possible.  This will ensure the proper gain setting change for the IV CPDS software modification.  A software upgrade is already in work. Support from the Software Verification Facility is being requested.

Update (6-12-01), based on the data obtained to date, it appears that the IV CPDS is either not responding properly or not receiving the correct command.  This type of problem has never been seen on the ground through SVF testing or ORT testing with the ground unit.  The problem described for the IV CPDS has some similarities to the problem described for the TEPC, though the signatures are not identical.  The IV CPDS is experiencing/responding to one of 5 possible failure modes: 

1. IV CPDS software bug.

2. IV CPDS hardware failure.  Likely candidates include the 1553 board or the memory board.

3. IV CPDS Power/Data cable failures.

4. Payload bus problem between PL MDM and UOP.

5. Payload MDM output problem.

In order to better isolate the problem, troubleshooting commanding and crew actions are requested as described below.  The steps below are described in chronological order:

1. Power cycle IV CPDS and command to Standby mode as quickly as possible.

2. If step 1 successful, command IV CPDS to perform a BIT test on the 1553 card and the memory card.  Dump the BIT data from the MDM for ground analysis. 

3. Connect MEC to UOP 2-J3 (UOP TEPC was plugged in to) and configure for 1553 communication per MEC – 1553 Card/Cable Assembly, Steps 1-6 (7A MEDICAL OPERATIONS).  As part of procedure, verify MEC is rebooted or powered on after removal of the LAN card. 

4. Perform mode commanding  to MEC (Standby to Acquire to Standby) at least 5 times over 48 hour period.  Verify MEC cyclic data command response echo indicates correct command was received.

5. Temporarily power off and disconnect IV CPDS from UOP 2-J4.

6. Move MEC from UOP 2-J3 to UOP 2-J4 and repeat step 5 above.

7. Power off and disconnect MEC from UOP and move it back to its original location, reconnecting MEC to the LAN (removing 1553 card and rebooting in the process).  Reconnect IV CPDS to UOP 2-J4 and power on

This anomaly is actually part of another anomaly documented in IFI #MER-0370. It will be worked as a part of that IFI. This IFI is closed due to replication of IFI 0370.

  • CREW EQUIP – (MER-0385)- TVIS Scraping Noise

Crew reported a rhythmic scraping/grinding sound in the front end, possibly on the left hand side. The sound is only heard at low speeds (2-3 mph) during motorized exercise. It is not heard at higher speeds nor is it heard at any speed when used in passive mode. Update (5-14-01) Troubleshooting procedure sent to crew, which included requests for video, photos and PC card data recorded during operation of the TVIS without a runner. MCC downlinked PC card data. Flight Project currently studying the all information media, along with crew comments (including stethoscope observations) to determine cause of problem. The crew reported that the noise appears to be coming from a section of the TVIS 2″ inside the transfer case and 2″ down from the surface of TVIS. They said it sounds like a diesel motor inside a diesel-powered ship.

Update (5-16-01): The data from the TVIS PC data card was analyzed and it indicated that the crew did not operate the TVIS above 7.5 mph as requested in the procedure. A report from the crew this morning confirmed that the crew commanded the TVIS to 10 mph but the data is not indicating that the TVIS responded. Also a sound wave file of the most likely motor box problem was uplinked to the crew for them to confirm it is the sound they heard. The crew called down this morning and reported that the sound wav uplinked was exactly what they were hearing. Update (5-17-01) Engineering has isolated the problem to a failed output phase inside the motor box. Further isolation inside the motor box is in work. The corrective action is to replace the motor box ORU with the backup unit. Further work must be done to identify the root cause of the problem to ensure it does not happen again. Update (5-21-01) Continue using TVIS in passive (non-motor driven) mode. Fly replacement motor on 7A.1

FIAR # JSC EA50123F has been generated to track this issue. This IFI is closed as of 06-15-01.

  • CREW EQUIP — (MER-0407) – Tissue Equivalent Proportional Counter (TEPC) Software Bug

Everytime a standby or acquire command is sent to the TEPC it does not receive the start command or any other command. It is not clear at this time whether this problem is location specific or not.

1. We are confidant that the TEPC commands from MCC through C&C to the input of the PL MDM is working fine. 

2. There are four candidates for the source of the TEPC problem. a) PL MDM intermittently outputting bad commands to TEPC (corrupted database); b) PL MDM to UOP bus is too long at some UOP interfaces leading to exceedence of 20 foot bus length limit; c) TEPC 1553 data card is intermittently failing; d) TEPC software has intermittent mode switching bug.  Possibility a is highly unlikely and b is moderately unlikely.  Either possibility c or d is the most likely problem.

3. Need to identify how many times mode command has been sent and how many times it has been executed, along with any other pertinent data such as power cycling and ground indications.  Further troubleshooting will be performed by commanding mode switches multiple times, possibly with the help of the crew power cycling.  In addition, a TEPC BIT test will be requested.  No further actions will be taken until this troubleshooting activity is performed and the data is analyzed. Update (5-23-01). Further investigations and TEPC command tests from GMT 2001/121 to 138 indicate that the problem lies within the TEPC software and not the PL MDM. Investigation is continuing.

Update (6/12/01), BME successfully commanded two BIT commands. Subsequent commanding caused the TEPC to hang. After hanging up, it was power cycled during its nominal relocation activity. Additional commanding is planned.

No permanent solution recommended until further trouble shooting is completed. The workaround right now is to have the crew revert back to the old way of doing the TEPC data transfer, which involves using the MEC and the TEPC connected via the RS-232 cable.

  • CREW EQUIP — (MER-0426) — BP/ECG Tripping Lab UOP#2

The crew was trying to do a Physical Fitness Evaluation (PFE) in the USL but the UOP the BP/ECG was plugged into (UOP 2) tripped its circuit breaker.  With some questions, we found out that the circuit breaker tripped when they turned on the UOP with the BP/ECG powered off.  They switched to the backup BP/ECG power cable and repeated the same problem. 

The crew tried several different troubleshooting steps and found out that 1) when the MEC is disconnected from the UOP, it does the same thing and 2) when the spare BP/ECG power cable is used, it does the same thing.  We verified that the RS-232 cable was not connected between the MEC and BP/ECG when the breaker tripped, so that was not a contributing factor.  Finally, the crew ran a UOP malfunction procedure that validated the UOP was functioning properly without BP/ECG connected.  I should also add that TEPC has been plugged  into this same outlet for the past month without any problems.

In investigating the UOP, there are two possible explanations for why it might trip.  First, it would trip if the ground leakage current exceeded 8 mA.  Second, if it has a high startup current spike, it would trip.  Since BP/ECG was powered off, and since it has a fuse in the cable itself, it is highly unlikely that it experienced a high startup current spike.  The more likely explanation is a ground leakage current problem, either from BP/ECG or the UOP limit being lower than expected.  Based on the BP/ECG ground certification data, it should not have leakage current over 0.1 mA unless it had some sort of unusual hardware failure.

Several things are in work.  First, we are working with the Robotics Workstation folks since they have had a similar problem on-orbit, and we might be able to apply their knowledge to our problem.  Second, we are contacting the electrical UOP experts to get some performance specifications on these UOPs.  Third, we are contacting the TEPC folks to see what their expected leakage current is for comparison to BP/ECG.  Update (6-14-01), An Anomaly Resolution Team (ART) was convened to work through what our next step should be with MER electrical and safety.  

Based on the sequence of events described above, the troubleshooting performed on-orbit, and the fault tree data, the following preliminary conclusions were made: a. The problem was not an MEC ground fault problem since the RS-232 wasn’t installed for all of the trip events.

b. There was not a short in the BP/ECG power cable since the UOP still tripped after the spare power cable was installed.

c. The UOP did not fail, but it is possible that because of the 2 ground wires in the BP/ECG plug, current leakage could be going through to the UOP and back to the BP/ECG causing the trip.

d. BP/ECG current leakage might be above specifications for the UOP GFCI, especially since the BP/ECG configuration tested on the ground (which included an inline resistor to simulate a human body) would have masked out circuit “noise” that would otherwise be present in the configuration on-orbit (since no human was in the loop).

e. As Crit-3 hardware, the BP/ECG was not required to go through the same level of EMI and other testing as Crit-1 and Crit-2 hardware.



No open IFI’s to Report



  • ROBO – (MER-0354) — Multiple MSS Cameras Routed to Same Video Line

After a series of video routing commands, the SSRMS Tip Elbow camera became routed to both channel 1 and channel 2. Then the SSRMS Base Elbow camera was routed to channel 1 also. This left 2 MSS cameras routed to the same channel. When all views were subsequently de-routed, the SSRMS Tip Elbow camera was left connected to channel 1.

A subsequent command to route outboard SSRMS cameras (Tip LEE) on channel 1 would not result in a video picture from that camera until the Tip Elbow TVC is de-routed from channel 1.

Event 13/16:25:00 Base Elbow camera routed to MSS Video line 1 in spite of Tip elbow camera already routed to that line.

Routings up to the event point appear normal until the following:

113/13:45:32 OCS> De-route Video Accepted

113/12:45:35 OCS> Route Video Accepted

113/13:45:43 SSRMS_Tip_Elbow_VDU_PFM_Source_Select_Video_1 – TVC

113/13:45:43 SSRMS_Base_Elbow_VDU_PFM_Source_Select_Video_1 – Upstream

Note: that the Tip Elbow TVC was already routed on MSS video 2 (since about 113/10:40) so the Tip Elbow should not have routed to MSS Video 1 (the VSUs should simply have routed the MSS Video 2 line to the 2nd destination) Proposed solution is to ask ISS to establish sequence of commanding events leading to this state and to determine whether or not the C&C MDM commanded a De-route for the Tip Elbow camera from Video channel 1 at 16:25 when the Base Elbow camera was routed.

  • ROBO — (MER-0382) – Latching End Effector (LEE) Temperature Ground Display 60 DEGC Higher Than Expected

Ground displays temporarily showed LMM and SMM temps of 60 degrees. Expected around 0 degrees. RATS-joint lee temps show a temp of 1 degree C. A telemetry problem is suspected. A spike was also observed from 1 to 60 to 1. Behavior of data indicates that the sudden 60 degree change in displayed temperature is not a physical issue since the thermal mass of the motors would preclude such a quick change in temperature. Therefore EVR believes there is no immediate safety impact to the LEE. No immediate impact to operations. ESC is analyzing ODRC data to determine the nature of the problem.

  • ROBO— (MER-0383) — Cupola RWS Monitors Did Not Display Monitor Numbers

After the Cupola RWS DCP was powered, the crew reported seeing the following on the RWS monitors:

MON1 – Black background with a Red & White strip with the words “No Sync”

MON2 – Black background

MON3 – White background

Monitor numbers were not displayed on any of the monitors. No video was routed to the monitors and the Cupola CEU I/O was inhibited. Shortly thereafter, the crew reported that MON3 changed to display “No Sync” and a black ground identical to MON1. Comm was enabled with the Cupola CE and everything appeared normal. A test pattern was then routed to all 3 Cupola monitors. The crew then reported that all three monitors displayed the test pattern and the monitor numbers were visible. There is a potential impact to robotic operations. Anomaly being investigated in RWS ART. Current focus is on a loose connector. Testing is planned at the MAIF; this will need to be scheduled to not interfere with the SSRMS s/w development and testing & the 8A s/w load. Recommend powering down both RWS units, check cabling, and re-apply power as part of an Airlock Dry Run activity. Investigation continues.

  • ROBO— (MER-0393) — R3D Wrist Roll Delta Transient

At 137:13:37:24 got R3D – SSRMS WR JEU Joint RDC BIT Fail” Robotics Advisory. This went into alarm and then to normal 10 seconds later. The alarm occurred during the extended joint range test, while in WR (Procedure 2.401, step 5, line 18 in Single joint table). The SSRMS safed automatically. Initial MSS Configuration: RWS CUP active, LAB backup; SSRMS Operational on Redundant Power String, Off on Primary Power String, RPCM closed to both strings to the SSRMS; SSRMS tip elbow, base elbow, tip LEE cameras powered on; All 4 SSRMS VDUs powered on; All 4 SSRMS lights turned off. MSS Operations: SSRMS operations in single joint mode; Coarse Rate selected; Rate Hold selected and driving motors at full rate (4 deg/sec). SSRMS WR was moving from -270 to +270; the RDC BIT failure occurred at -9. Since this alarm cleared within 10 seconds, there is no identified impact to operations at this time. Anomaly is currently low priority and does not prevent arm motion. PRACA PR# 2705 was generated to track this issue.

  • ROBO— (MER-0394) — SSRMS FMS (Force Moment Sensor) Calibration Aborted

At 137:16:11:41 after sending an FMS Calibrate command got “Calibration FMS Aborted: FMS Calibration Failed” discrete message. Then received approximately seven R7B alarm messages that went to norm. The command was re-attempted with the same results. Effects ability to use Force Moment Sensor SSRMS capabilities. No near term impacts. FMS capabilities are not planned for use until TBD. Anomaly is currently low priority and does not prevent arm motion. Trouble-shooting tests were performed on 5June01 and FMS calibration on the Primary power string was successful.

  • ROBO— (MER-0395) — Redundant String ACU (ARM Computer Brake Unit) Brake Voltage Errors During Brake Release

At 137:14:24:16 got “R3B – SSRMS ACU DPRT Brk Switch Fail”, “R3B – SSRMS ACU DPRT Brk Voltage Fail” and “R3B – SSRMS Brk Voltage Err”. The alarm occurred during the extended joint range test, while in WR (Procedure 2.401, step 5, line 18 in SJRM Table). The SSRMS safed automatically. Initial MSS Configuration: RWS CUP active, LAB backup; SSRMS Operational on Redundant Power String, Off on Primary Power String, RPCM closed to both strings to the SSRMS; SSRMS tip elbow, base elbow, tip LEE cameras powered on; All 4 SSRMS VDUs powered on; All 4 SSRMS lights turned off. MSS Operations: SSRMS operations in single joint mode; Coarse Rate selected; Rate Hold selected and driving motors at full rate (4 deg/sec). SSRMS WR was moving from -270 to +270; the ACU errors occurred at -30. Crew could not subsequently remove brakes, without the error re-annunciating and the SSRMS re-safing. May require EVA for ACU R&R. May require power supply to SSRMS on the Primary Power String for all SSRMS ops. May have non-compliance to flight rules and safety requirements, which state that 2 power strings must be operational prior to execution of assembly operations. Testing conducted to repeat failure signature and gain a consistent signature. Failure signature was repeated first 3 times however subsequent 7 times signature not present (nominal signature). Have tried to repeat failure signature several times with no reoccurrence.

  • ROBO — (MER-0406) – Cupola RWS Auto Sequence Pause/Proceed Switch Xcheck Anomaly

During a joint OCAS, R1H – MSS CUP OCS DCP Sw Fail was annunciated due to an Xcheck error of the auto sequence pause/proceed switch. This failure dropped the active (Cupola) RWS to backup. The switch was cycled 5 times in each direction (proceed/pause) to gather data. The first two attempts to apply Pause resulted in failures but all other switch throws produced no OCS errors. Initial analysis of REM log indicates that during initial throw of the proceed switch contact closure was less than 1 second in duration because it did not show up in the 1Hz data. Current focus is on operation of switch throw. Suspect differences between FEU units on ground and flight unit have contributed to possible switch throw issues. Investigation continues.

  • ROBO — (MER-0409) — Shoulder Pitch JEU Anomaly

  • Received R3C SSRMS SP JEU Transmit and Receive Errors while SSRMS operational on redundant string, but not being used. JEU was rebooted, and failure was re-annunciated. Expect loss of SSRMS redundant string until joint R&R (if not resolved). Troubleshooting conducted that included a software patch. Focus is on potential JEU 1553 address problems. MDR developing a diagnostic patch that will identify just what is happening. This is being worked in the SSRMS ART on a daily basis. PRACA PR# 2706 was genearted to specifically track this problem.

  • ROBO — (MER-0419) – Cupola DOUG Application Not Working

DOUG was activated by Alpha crew to provide complementary situational awareness in support of SSRMS ops from the cupola RWS on 6/05/2001. To provide a DOUG-enabled SSC client for the cupola RWS, Susan installed an SSC client load hard drive in her personal laptop. Upon reboot with the SSC hard drive, the BIOS reconfig tool AutoXD detected a BIOS change and prompted Susan of this, asking her if she wanted to update the BIOS to correct the perceived problems. Apparently, she instructed AutoXD to perform the reconfiguration. As a result, the serial port (comm1) on her SSC was rendered inoperable. This resulted in an inability of the PCS-DAS program to open the serial port to receive data from the PCS. Susan recognized this from status messages output by PCS-DAS and called MCC for support. She then retrieved another SSC and connected it to the cupola RWS PCS via the serial cable. PCS-DAS was activated and did not report problems opening the serial port. She then started DOUG which successfully connected to the PCS-DAS server. SSRMS joint angles of all zeroes were observed in DOUG. Susan attributed this to the current SSRMS status (arm in keep-alive) and expected that when the arm was powered up, PCS would send the joint angles to DOUG via the serial cable and PCS-DAS.

Some time later, Susan attempted to use DOUG to familiarize herself and Jim with the expected SSRMS configurations that they would see during their upcoming maneuver. She used DOUG’s JntSystems->SSRMS dialog to input joint angles while DOUG was still receiving data from PCS-DAS. When the DOUG SSRMS model did not move to reflect her manual joint inputs, Susan called MCC for support. She was informed by DOUG support that while DOUG is connected to PCS-DAS, manual joint angle inputs would not be recognized. Susan was presented with the option of restarting DOUG in a mode that would not connect to PCS-DAS, allowing her to manually input SSRMS joint angles that would not be overwritten by telemetry. Apparently, she did not exercise that option.

As SSRMS ops from the cupola RWS commenced, Susan reported that DOUG was getting no joint data from PCS when the arm was powered up. She was instructed to shut down DOUG and PCS-DAS and then restart PCS-DAS and DOUG (in that order.) Susan reported that DOUG was still not getting telemetry from SSRMS. SSRMS operations were completed with DOUG receiving no real

time telemetry. After SSRMS operations were completed, Susan expressed an interest in continuing DOUG troubleshooting efforts. DOUG support indicated that there was an application on the SSC that would aid in this effort and that procedures could be developed to help her perform the troubleshooting. Susan asked DOUG support if she could continue troubleshooting at that time. Susan was then stepped through the use of the hyperterm application on the SSC. Results of troubleshooting with hyperterm revealed that the SSC on which DOUG was running was not receiving any data over the serial cable. This narrowed focus to one of two possible sources for the lack of data: a bad cable, or no data from the PCS serial port. Susan and DOUG support agreed that the next course of action should be to take the serial cable from the lab RWS and reinstall it in the cupola RWS. Upon completion of the installation of the lab cable in the cupola RWS, no data was seen on the SSC serial port. Susan then suggested going back to the lab RWS and trying to activate DOUG there, commenting that it had worked properly there once before. DOUG support concurred. At this time, Alpha entered an LOS situation and comm was lost. During the intervening minutes in LOS, Susan performed several reconfigurations to try and isolate the loss of data fault. Upon AOS, she reported the results of her troubleshooting. The following table summarizes her findings.

Config Result:

lab RWS with lab RWS serial cable and lab RWS SSC – good data

lab RWS with lab RWS serial cable and cupola RWS SSC – good data

lab RWS with cupola RWS serial cable and cupola RWS SSC — good data

Susan proposed and DOUG support concurred that the evaluation indicated two good SSCs, two good serial cables, a good PCS serial port on the lab PCS, but a bad serial port on cupola PCS. There is no impact to the vehicle.

PCS personnel (Petr Polak) had the crew download the data file for the System IRQ Assignment and found the following on the Cupola PCS:

Serial-A (3 or 4) marked as “X”, it should be marked as “O”. This error in configuration would prevent the data transfer via the serial port on the computer. O= Currently assigned; X= Will be assigned after Enable is selected.

The PCS Hardware/OS folks looked at all the other CMOS settings and found no other abnormal values.

Since the CMOS Check procedure was run completely, the CMOS reset was performed, then the PCS should be in nominal configuration for operations. Investigation continues.

ROBO — (MER-0420) – PFM Carrier on SSRMS Video Line Flicker

On day 113 between GMT 18:37 and 18:43 we see the PFM_Carrier_On_Video_2_Output_SSRMS_Base_LEE flicker “present” and “absent” even though no video is routed.

Again on day 156 between GMT 15:55:12 and 16:00:19 we see PFM_Carrier_On_Video_1_Input and _Output_SSRMS_Base_LEE and the PFM_Carrier_On_Video_1_Output_SSRMS_Base_End_Elbow statii flicker “present” and “absent” again when no video is routed.

Around day 162/20:45 PFM__Carrier_On_Video_1_Output_SSRMS_Base_LEE is showing solid “present” status when no video is routed. Currently investigating possibility that open end of video lines (at Tip LEE) are providing an antennae for EMI and this is being registered by PFM monitors in the VDU.

This behavior is unexpected and currently unexplained.

Currently investigating possibility that open end of video lines (at Tip LEE) are providing an antennae for EMI and this is being registered by PFM monitors in the VDU. Investigation continues.

  • ROBO — (MER-0438) – SSRMS Load Cell Drift

Under Investigation



No open IFI’s to report


SpaceRef staff editor.