October 30, 2007 notes:

 

We failed.

 

I could try and put the best positive spin on it, but the bottom line is that we failed to win anything in the Lunar Lander Challenge at the X-Prize Cup this year. We made two perfect flights, and came within eight seconds of winning, but we also had three engine-damaging hard starts.

 

Just like last year, this is a quick media-free update. Matt will be doing a separate update with all the video when he gets back from another trip out of town.

 

There was a lot of stressing in the couple months before the cup, but it was mostly over not having multiple backups for some of the things we were taking, and we thought we were just about as prepared as we could be. We had already completed the full level 1 requirements under experimental permit at the Oklahoma spaceport earlier this year http://media.armadilloaerospace.com/2007_06_03/LLC1demo.mpg , we had done six tethered flights of three minutes each, and we had literally dozens of other successful tethered flights.

 

Crashing Texel two months ago was a blow, but the modules were essentially flight ready, so we rushed the permits ahead for them. It is worth noting that we did not have the paperwork done at the time that would have required AST to reply before the cup – they would have been perfectly within their rights to give us the permit a few weeks after the cup. We bitch and moan about AST as much as anyone that deals with them, and I think we have plenty of legitimate gripes, but the people we are working with very much want to see us succeed. We noticed that we were trading permit related email with them on nights and weekends, which isn’t the normal image of a clock-punching federal employee.

 

One issue that we were pitching a fit over was that somehow our rather expensive flight insurance policy was deemed invalid for experimental permit work. After they had let us fly with it earlier this year. Two weeks before the cup. Phil took on the work of getting it all straightened out, which mostly involved separating out our non-flight insurance and arguing over the validity of some industry standard exemptions. If you need rocket flight insurance, you should be talking to Ralph Harp rharp@falconinsurance.com, he has gone above and beyond what we could have expected to help us out.

 

We were required by AST to do an untethered free flight of the module at the Oklahoma Spaceport before flying it in front of the crowd at the cup. We would have preferred not to risk the vehicle, but I very much approve of actual demonstration over yet more paperwork for convincing regulators, so I didn’t argue too much. The OKSP people are always a pleasure to work with, and the flight went perfectly: http://media.armadilloaerospace.com/2007_10_21/modFreeFlight.mpg

 

Right after we crashed Texel, I had put in an order for another spare Crossbow FOG IMU, and was disappointed to get a 14 week lead time for the replacement part (the FOGs now come from a Russian company). I didn’t worry about it too much, because the IMU from Texel’s box still seemed to be working, so I planned on keeping it as a backup to the main, never-before-crashed electronics box. When we were doing final backup checking in the few weeks before the cup, I found that while that IMU was spitting out data, one of the gyro axis was completely non-responsive. I really didn’t want to go to the cup with only one functional electronics box, so we started scrambling to work a backup.

 

Plan A was to see if Crossbow had ANY other FOG based unit in stock, even if it was a demo unit or one of the VG or AHRS models. Plan B was to see if Crossbow could repair the unit in time. Plan C was to resurrect the very first Crossbow IMU we used, which was somewhat lower grade, and required slightly different software and a different power supply voltage. Plan D was to combine the Texel crash IMU with one bad FOG and the IMU from our last peroxide vehicle crash with two bad FOGS to get three functional FOG axis. Plan D was to resurrect the analog output KVH FOGs that we used before we moved to Crossbow units. Crossbow checked for any units on hand, but they didn’t have any. I talked to their repair tech, and he said that they wouldn’t be able to fix anything for a couple weeks because their bake-in ovens were completely booked with new parts. Ah ha! All the new gyros had just arrived, so they were able to skip a bit of final qualification and ship my original order out the next morning. We also implemented Plan C, so we had three complete electronics boxes to take to the cup.

 

One of the most distressing events was that one of the axis drivers on our Haas trunnion table died a couple weeks before the cup, preventing us from drilling injector holes. It reported a cable fault, so we hoped that it would be an easy fix, but after calling it in to our local Haas dealer, the days ticked by with no tech arriving to look at it. Our daily (or more often) calls had them telling us we were eighth in line for service, and they will get there as soon as they can. We asked them to see if one of the service people would take overtime pay to come in on Saturday, and they agreed. The tech checked the system and said there was nothing he could do, and that we would have to send it back to the factory for repair. There were no demo or rental units available. We would up taking it apart ourselves to see if there was anything we could do, but the cable seemed to be good all the way up to the deepest guts of the machine, so we finally boxed it up and shipped it fed-ex off to Haas for repair. After telling them about the time-critical nature of the problem, we got it back in short order. When we plugged it all back in, it gave exactly the same error. We got the local Haas dealer to send out a different tech, and he replaced a board in the mill, which got us back running again.

 

We still had three sets of gimbal actuators, but due to some issues that will be discussed below, we were concerned that we might break some in the final weeks before the cup. The head of http://www.ultramotion.com/ has been very supportive of our efforts, and had two extra units rushed out to us.

 

Tex Air Cryogenics http://www.tex-aircryogenics.com/ has been our lox supplier since we moved away from peroxide, and they have really become personal friends as well. We were still making changes in how we planned to measure and load lox for the level two flights in the last couple weeks, and Reid was over at our shop long after dark personally installing new dewar equipment. Since they were the official supplier for the cup, we were a little worried that their might be some charge of favoritism leveled, but the lack of any other competitors made it a moot point. After the lox truck and the helium truck had left, Reid loaded up a final batch of parts for us into his pickup truck as he was headed out for New Mexico.

 

Now we get to something that foreshadowed the outcome. We have gone through at least a dozen graphite engine chambers in the last year as we have experimented with different jacket and injector designs. The team at Cesaroni Aerospace http://www.cesaronitech.com/ has given us excellent service all year, but as the cup approached, they would machine an order from us the very next day, and get it shipped out for next day delivery. Many of our chamber retirements were due to an injector problem getting hot oxygen onto the graphite throat, resulting in an uneven erosion (we had a few last-ditch chambers still on hand), but several were due to cracks from startup transients, and a couple were more thoroughly destroyed. We wanted to have four pristine chambers of the final form on hand.

 

To reduce the worst-case failure distance that one of the modules could fly, we had decided to shrink the nozzle throat from 3.25” to 2.75”, giving a significant reduction in thrust. We had a few good flights with this configuration, but we also had startup difficulties and eventually two hard starts that completely destroyed engines. Our theory was that the reduced throat caused more back pressure in the chamber during startup, pushing hot gox farther back into the cooling jacket before fuel filled it. We eventually got AST to agree to let us use the same big-throat engines that we had been flying all year, on the condition that we carry twice the required payload mass to ballast it down a bit. We were happy with this, and with the Haas mill working again and more graphite on the way from Cesaroni, we had four engines we could use at the cup.

Arrival

 

As a competitor, Holloman AFB turned out to be a great place to work. We had better support and facilities than we had at Las Cruces last year, and we really didn’t have any issues. We, and the event in general, had the full support of everyone from the base commander on down. For instance, there was an awkward regulation in place that prevented us from using our crane when the truck had both fuel and oxidizer on board, so moving the vehicles around and standing them up was resolved by having our base liaison just grab a half dozen convenient enlisted men to heft things around by hand.

 

We had asked the chief judge for some specific rulings when we arrived – would we be allowed to fly the backup module as-is if necessary, versus having to swap parts over to the original module, and would we be allowed to fly Pixel for level one if we destroyed both modules. We were very happy to get a yes to both of these on Thursday, but both decisions got reversed on Friday. The backup vehicle bit was specified in the contestant entrance agreement (but not in the prize rules), so there wasn’t too much we could say about this, but we had been told on the ground by the judges last year that we could fly the level two vehicle for level one if we were willing to relinquish the chance for level two, so we were a bit upset about the change. I didn’t want to even try for level two without having claimed level one. The official ruling was that the tanks determined the airframe, and we could swap everything else from the backup vehicle if necessary. While it turned out not to make any difference, I don’t agree with the thinking that doesn’t allow backup vehicles, since the point is to Get The Job Done, and backup vehicles versus backup systems is a completely valid direction, and NASA had nothing against it in the official prize rules.

 

The pad survey went fine, and we double checked everything to make sure our new guidance algorithms were dealing with it correctly this year. We had landed fine on the 8 meter octagonal pads in Oklahoma, so 10 meter circular pads shouldn’t present a problem. The new pads were flush with the ground, which prevents a tip over like we had on one of Pixel’s flights last year, but it also made them harder to distinguish. The lunar pad was more challenging than last year, with bigger rocks, deeper craters, and no clear spot directly at the center.

 

Saturday

 

We rolled across the starting line at 8:00 AM sharp, and headed for the pads. Everything went smoothly through the checklist, but the engine failed to start. Several tries kept resulting in an immediate SHUTDOWN_IGNITER, because no significant pressure was being generated, and it wasn’t safe to open the main propellant valves.

 

Investigation showed that the fuel orifice on our igniter was almost completely clogged, barely letting anything through. Russ had noticed that the igniter bleed step in our checklist had taken much longer than expected, which should be a good warning sign. We couldn’t find any 0.030 wire to try to push through it, so we went back to our hangar and put our pressure test plate on the bottom of the engine to allow us to back-flow high pressure helium through the engine with the igniter solenoid removed to clear it. This seemed to allow it to flow normally, so we buttoned everything back up for our next try.

 

At 12:45 we headed out again. Everything went smoothly again, and we had a perfect ignition and flight. Guidance and control was perfect, letting us burn the X-mark off the target pad. Landing was perfect, with an instant engine shutoff based on the ground contact shock signal. When Russ looked under the engine, he did find that the igniter was somewhat eroded, which was unexpected, but it should still work. We refueled for the return flight, but during the igniter bleed we found that it was again almost completely clogged. That likely explained the igniter erosion, as the normally-rich igniter had probably crossed through stoichemetric as the orifice clogged up again. We were way ahead of the clock, so we had at least a half hour to try to resolve the issue. What eventually worked was a paper clip manually filed down to fit in the orifice. Once we had good flow, we finished the propellant loading and got ready for the return flight.

 

The engine started with enough of a bang to cause the vehicle to actually hop off the ground, but the engine was still lit and attached to the vehicle at the end of the start sequence, so I lifted off to a few meters altitude. From the exhaust plume it was clear that the engine wasn’t very healthy, but I decided to go ahead and go for it. I hustled the vehicle up to altitude, over to the second pad, and down towards the ground as fast as possible, so if anything happened, it might be a survivable fall. There were some very disturbing impulse events during the flight over, which were probably pieces of the engine departing the vehicle. I was letting the clock run out only a few feet above the ground when, with only a few seconds remaining, something (probably half of the nozzle retaining ring departing) pushed the vehicle hard enough to cause it to tilt past the abort limit even with the gimbal scrambling over to maximum compensation. The vehicle tilted over and fell to the ground on its side. Total flight time: 82 seconds.

 

It turned out that nothing was really hurt on the vehicle besides the engine chamber, and we got the vehicle back up and offloaded without any problems. Much to our surprise, none of the shock-watch stickers on the tanks had tripped, meaning that the body had taken less than 25 G shock. While it was still functioning fine, the electronics box on top had a handle bent in significantly, and both the 25 and 50 G shock stickers had been tripped, but not the 100 G. I always expected the computer box to take a lot less shock due to the wire rope isolators, but this side landing seems to have whiplashed it into the ground even harder. We moved this box to backup status, and planned on using the one from Pixel for the next attempt.

 

We chalked the hard start up to some combination of the eroded igniter or the manually reamed out orifice, and focused on the repeated clogging of the igniter orifices. We had never seen this before, but Paul Breed of Unreasonable Rocket http://unreasonablerocket.blogspot.com/ had reported very similar problems with his igniters. The Breeds reminded us of their issues, and they gave us one of the sintered filters off of their display vehicle to use. This is quite heroic sportsmanship if you think about it – it looked like we had everything else well in hand, and resolving the clogging problem was very likely going to result in one of the prizes that they were competing for being taken off the table. We gave them as in-depth a look at our hardware and operations over the next couple days as possible to try to return the favor.

 

We put a backup engine and computer back on the module, and with the filter installed and the brand-new igniter, the flow during the igniter bleed was clearly better than the previous one had been all day. We still don’t know the exact cause of the clogging – it might be the different ethanol that was delivered to the base, it might have been contamination in the fuel filter that was just added to the fill hose (we removed it before the second flight), or it might have been something kicked loose internal to the vehicle over the thousand miles of transit from Dallas.

 

Sunday

 

We again rolled out at 8:00 am sharp. The vehicle again made a perfect flight to the center of the second pad. The startup had been with a bit of a bang, but when Russ checked the chamber, it was still fine. The igniter was perfect, so we felt that the filter had prevented anything from clogging it, and we expected the next flight to start fine. During pre-flight, the igniter bleed was completely unchanged, so we were very confident that aspect was resolved. Ignition was with a severe bang, and when I lifted off, it was clear that there was even more significant damage to the engine than on Saturday, with a lot of raw fuel coming out the bottom. Rather than risk it falling out of the sky at 150 feet, I just set it back down on the pad for an intact-abort. On inspection, the chamber was again shattered, and a large chunk had already been ejected out of the nozzle, so various burn-throughs of the aluminum would have been inevitable. Not going for it was the right call.

 

Our best guess for this hard start was that we were going significantly faster between flights than we usually do at our test site, and there might be more fuel remaining in the jacket or soaked into the graphite than we have at our normal tests. We decided that we would do a significant helium purge between runs, and possibly just sit around for a little bit before doing the return flight.

 

We put yet another new engine on the module, and headed out for the 12:45 window. Everything went perfectly smoothly up to ignition, then we had the most violent hard start we have seen. The engine jacket was ripped off the injector, and the graphite chamber was completely shattered and thrown over a wide area. We were done. We invited AST out to the pad to inspect the aftermath, and we carefully noted where all the pieces had gone. We found a good sized chunk of graphite stuck in the ground 64 meters from the center of the pad.

 

We had run out of easy explanations for the engine starts. This was a brand new engine, so it could have been some assembly or fabrication error, but we couldn’t have been so unlucky as to have three completely different causes of hard starts in two days. We think it is likely that our engine design has always been just on the hairy edge of ignition problems (the problems with the 2.75” throat versus the 3.25” throat seems to imply that), and some subtle difference in conditions between Dallas / Oklahoma and Alamagordo was just pushing it over the edge to a three-hard-starts-in-five-tries level. The altitude might have something to do with it. A one psi difference doesn’t sound like much, but when I opened the top of my toothpaste at the hotel, it started extruding out all by itself. Another rocket developer also commented that a seemingly reliable engine developed problems after a move to a desert location. The lack of humidity might have played a part, making it easier to ignite something in the cooling jacket. The ethanol might be slightly different. We just don’t know.

 

Postscript

 

There isn’t anything I can look back at and say we did obviously wrong. We had backups for everything, we had demonstrated the required flight performance many, many times, and we had made three free flights under experimental permit since the last XPC, all without incident. The problems we had at XPC last year were clearly solved – our landings were accurate, and our landing gear didn’t break. I am honestly very surprised that we didn’t take any of the prizes. My final estimate before we made the first flight was 90% chance for level 1, and 60% for level 2. There were a lot of things that branched off of the “if we win the LLC” path that we aren’t going to be able to do now.

 

It is possible that we can fix up the ignition with some simple changes to the purging, but at this point I am inclined to make some drastic changes to the engine design. I had already been worried about in-air restarts when coming in for a landing with low tank pressures, so we now have more than enough incentive to make starting absolutely reliably a top engineering priority. At least two of the hard starts clearly happened inside the cooling jacket. We have a much larger volume jacket than is conventional, enabled by the graphite chamber not actually requiring all that much cooling. This gave us the benefits of very low pressure drop and a lack of sensitivity to minor tolerance issues, but it also caused the start sequence to take a lot longer, and provides a lot more volume for a combustible mixture to form in. We could go back to an uncooled chamber like we had last year, or possibly even a simple stainless chamber with heavy film cooling. I don’t think we could make a 180 second flight with one of those engines, but practically everything else that we care about would still be fine. We have learned a lot of things about igniters and sealing in the last year that would allow us to make that class of engine run better than it did for us before. For the regen engine, I am going to investigate moving the throttle valve after the cooling jacket to eliminate the need to fill all that volume. The raw graphite would leak fuel through the pores, but we have been talking about chrome plating the chambers to avoid oxidation anyway. A face-shutoff pintle is sounding even more inviting from a dribble volume standpoint.

 

None of the vehicle frames are hurt, but we took back three wrecked engines, and mod1 needs a completely new wiring harness. We are going to take a little time off to recharge, and when we get back to it, we are going to have to give some more priority to the Air Force and NASA contract work. I haven’t read any of the event coverage yet, or even gone through my rocket related email yet, because I am still sulking a bit.