NASA Ames CIO Blog: Behind The Scenes of The ISS Photosynths

Last week’s launch of the new International Space Station (ISS) Photosynths was the result of 18 months of dedicated work by a great team of people across the Agency and at Microsoft. Having played a role in facilitating the collaboration, I thought I’d invite some of the team members here on my blog to share their personal recollections of how it all came together. The project serves as a good example of how NASA succesfully collaborates with private industry to innovate ways of bringing space exploration to the public. Reading through the coverage on some of the tech blogs here in New Media Valley, there seems to be a healthy appetite for more innovation in this space (e.g. TechCrunch, Gizmodo, Ubergizmo). Adam Sheppard, whom I first met back in 2006 when he was working with the Photosynth Team at Microsoft Live Labs, offered to start off by sharing his experiences working with NASA on this project.

By Adam Sheppard

In October 2006 I was sitting in my office at Microsoft trawling through a backlog of email and voice messages. It had been a busy couple of months leading up to the Technology Preview of Photosynth at San Francisco’s Web 2.0. We’d brought the house down with our on stage demonstration of Photosynth’s ability to take a large number of digital images and automatically assemble them into a high resolution, 3D environment that anyone could explore at home from within their web browser.

Around the same time, Chris Kemp (Currently CIO at NASA Ames) had just joined the Agency and was leading their business development efforts to seek out new and emerging technologies coming out of the private sector that could help NASA’s mission. He had seen the Photosynth demo and was eager to learn more about what he’d seen. He was full of ideas and saw the potential for Photosynth to bring the public closer to the space program than they had ever been before. They could follow in the tracks of the Mars Rovers, see every nut and bolt of a shuttle on the launch pad and experience first hand life onboard the International Space Station (ISS). At Microsoft Live Labs we were bringing some of the top computer scientists in the world together with talented engineers to explore new ground on the web. We couldn’t have asked for a more kindred spirit than the men and women of NASA who, through their own genius and engineering skills, were exploring our universe for the benefit of mankind.

Throughout 2007 we began planning and experimenting with images provided by NASA. Together with Microsoft Researcher Drew Steedly (one of our lead scientists on Photosynth) we visited Kennedy Space Center as preparations were underway for Shuttle Endeavor STS-118’s mission to the ISS. Chris had arranged unprecedented access to for us to photograph and document the Shuttle as tiles were being replaced, in the Vehicle Assembly Building, and on the launch pad itself. We were really eager to get some aerial shots of the shuttle awaiting launch, and Drew was lucky enough to sit with a SWAT team in one of those ‘Men In Black’ NASA helicopters as they flew a security fly-by around the shuttle on the pad, producing some amazing 360 degree shots we would later include in the synth. (video)

For the launch itself, we sat with the families of the astronauts, and one couldn’t help but remember the footage of the Apollo missions with people watching from the same bleachers as their loved ones journeyed to the Moon. A few days later, back in Seattle, we were proudly sharing our unique stories with our colleagues, when a call came in that had us jump into action. Since the Columbia tragedy occurred three years prior, many new safety procedures were instituted to help ensure the safety of the crew. One such measure was the detailed inspection of the underside of the shuttle from the ISS using both photography and a laser scan to assess any potential damage. Some damaged tiles had been found on Endeavour while in orbit and NASA was scrambling to make a decision on whether they should attempt a repair. During our previous discussions with NASA engineers, the idea of using Photosynth for safety procedures had been discussed and this was the perfect test case for us. Each tile is unique, carefully cut and individually serial numbered before being adhered by hand to the underside of the vehicle. Typically a specialist would have to trawl through thousands of close up images, cross-referencing against plans to visualize where on the shuttle the particular tile was. To help the specialists, we took all the images downlinked from the ISS while it was still in orbit, and used Photosynth to reconstruct the bottom of the shuttle, automatically placing the images together. We were all amazed at the results as we zoomed into the damaged tile. And while the decision had already been made to proceed without a repair, the value of this new technology had certainly been demonstrated.

Shortly afterwards I found myself at NASA headquarters in Washington DC demonstrating the synths to the Associate Administrator for Space Operations, William H. Gerstenmaier. He immediately saw the potential to share life onboard the ISS using Photosynth, and we soon found ourselves on our way to NASA’s Johnson Space Center in Houston to help train the astronaut on creating a full-size synth of the International Space Station. (video)

As we worked with the astronaut trainers, we soon realized that there were some unique challenges to consider in zero gravity. For example, Photosynth makes some assumptions on which way is ‘up’, but when you’re floating in space there is no ‘up’. With barely enough room to stand up, and every available space full of equipment, we also had to consider what technique would best capture both the panoramic views and detail shots that would lead to a successful synth. Towards the end of the day we got to meet some of the astronauts being trained on emergency procedures on a life-sized mockup of the Shuttle and finally had the chance to sit in a real Russian Soyuz capsule also used for training.

Needless to say it was my life long dream to be crawling through a space station and meeting the men and women who work tirelessly behind the scenes at NASA.

I personally moved on to other projects mid 2008, just when the flight plan for the new ISS Photosynths was coming together. All I can say is the resulting synths have turned out better than I could have ever imagined. I’m delighted to think that somewhere there’s a child exploring them right now who will one day walk on the surface of the Moon or even Mars.