October 23, 2006 notes:

 

X-Prize Cup 2006

 

It has been a month with lots of drama and excitement, but unfortunately, no prize money. Matt will make a separate update later with lots of pictures and video, but I wanted to get all the events down while they are still fresh in my mind.

 

The five hour drives each way to Oklahoma for flight testing on Saturday were really wearing on us, so we went ahead and did some fairly long hover tests right at the shop. Our immediate neighbors know all about what we do, and don’t mind – we started taking a radio over to the night watchman to alert him before we did something really loud. This time, we did manage to get someone else to lodge a complaint against us, causing our landlord to give us a formal cease-and-desist letter, and sending the fire marshal out to our shop. Honestly, we deserved it. We got away with a lot at the shop over the past five years, but I can’t blame anyone for calling some of our later tests unacceptably loud and long at that location.

 

The fire marshal didn’t really find anything seriously wrong with our operation, except that we were over the storage limits for LOX and ethanol, which was only due to the run-up in testing before the XPC, and we didn’t have a welding permit for our newest welder. The big problem came when they looked at our occupancy permit, and we were all shocked to find that we were classed as “office space” instead of “light industrial” like the rest of the building. They shut us down and had us vacate the premises. Two weeks before XPC.

 

Phil took the next day off from work and scrambled around to talk to all the right city folk and get all the paperwork filed properly. We moved our lox to outside storage and our fuel to our remote test site, and we were able to get back to fabricating in the shop in a couple days.

 

We just couldn’t travel all the way to the Oklahoma Spaceport for each of our remaining tests, and our negotiations to move operations to a local naval air field weren’t going to complete fast enough, so we decided that we had to go back to the 100 acre property in Rockwall. We had abandoned testing there after a neighbor complained that we had scared his (very expensive) horses badly enough that they jumped a fence. We didn’t want a feud with anyone, because what we do is unusual enough that I’m sure someone could find some regulation to shut us down with if they really wanted to. With only a week and a half remaining, we tried to cover all the bases – we talked openly with the city, the police, and the fire department to make sure we would be above board (it turned out the only restriction was noisy operations finished by 7 pm), and we went around to all the neighbors personally briefing them on what we were doing. We paid to have a police officer and the volunteer fire department present at the tests. In the end, it all worked out fine, and the neighbors didn’t hate us afterwards.

 

We found a couple issues of concern in our local tests:

 

We were nearly saturating the bandwidth of our telemetry link, causing some latency issues. Adding some new real-time graphs for packet arrival jitter and latency were very helpful in understanding the problem. I was able to adjust the link bandwidth, trim some data, and work around some poor Win32 timer performance issues on the laptop to fix it.

 

The 1.2 ghz wireless cameras were occasionally causing interference with the GPS. Several times I had seen the GPS PDOP drop to 0 for five seconds during ground operations, but it had never happened during flight (it would have triggered a soft abort, with an attempted touchdown based strictly on the IMU). We positively tracked this to the cameras, but Russ believed the cutouts were only happening when we had everyone and the truck milling around the vehicle, multipathing everything.

 

We finally did definitively resolve our engine injector melting problem. We have had great combustion efficiency with the two-row twisted unlike injector pattern, but we continued to erode parts of the injector. Stainless steel versions lasted longer, but they still suffered. We did make two back-to-back 110 second flights with a single stainless injector, but it was very close to burning through at the end. The problem turned out to be quite simple – you just can’t cool a significant part of the injector face with lox, at least when it is at sub-critical pressures. Phil had brought this up a couple times in the past, but we had continued to think that we had some kind of flow recirculation problem instead.

 

The fix involved shrinking the lox injector ring from 1” wide to ½” wide and letting the lox distribute in a separate manifold before flowing down into the ring, rather than flowing around the ring. The current implementation of this has a single weld between the fuel manifold and the lox distribution manifold, which I am not happy about, but we will be redesigning this in the future to avoid it. This is our 20^th injector design at this size, but it looks like we finally got it – very high performance, and absolutely no erosion or heat damage, even in aluminum.

 

Unfortunately, now that the injector was no longer melting, we found the next weakest link. After 90 second flights, the carbon wrapped graphite chamber (fabricated by Cesaroni Aerospace) was glowing a dull red on the outside above the converging section, fraying up broken fibers from the ground outer surface, and we were finally getting a little erosion in the converging section of the nozzle. This didn’t worry us too much, but when we loaded up for a 180 second flight, which involved a 50% higher chamber pressure, at 52 seconds into the burn, the orange-hot chamber shattered into a million pieces. There wasn’t any overpressure noted on the pressure transducer, and the transducer was still functioning afterwards, so the chamber must have just gotten hot enough to cook off all the external binder in the reinforcement and oxidize away much of the carbon fiber, then some minor excursion shattered it. It was an impressive fireball, but all it did was strip off the insulation on the inside quarters of the lox tanks and burn a couple wires.

 

We made a quick change to double the size of the film cooling holes, from 19% of the injector area to 34% of the injector area, which will certainly hurt the Isp, but would probably still let us (barely) make a 180 second level 2 flight. We didn’t get a chance to try it. We did a quick tethered test with the modified engine that showed it still worked fine, then we went on to a critical test we still hadn’t done: ground liftoff.

 

The first attempt at ground liftoff showed two problems: it didn’t lift off, because the vibration at the ground caused enough noise in the accelerometer to bounce it above and below the desired acceleration level, keeping it from throttling cleanly up. We have seen this exact problem on previous vehicles, and the fix is easy – just force full throttle, or something close to it, for the first second or so of flight. I made a trivial change to the software to go for a higher liftoff velocity, but we didn’t have enough helium to re-pressurize for another try, or enough time to go back and get more. The other problem was that the engine heat at liftoff (well, failure to liftoff) took all the temper out of the springs on our landing gear. It might not have been a problem if the vehicle had lifted off cleanly, but after we replaced the springs we fabricated leather “rocket socks” to protect the springs.

 

Then it rained all day, and we weren’t able to test the next two days before we had to pack and head out to the show.

 

At this point, we still didn’t have our launch permit to actually fly at the show. The final, last minute wrangling with AST involved us defining and performing a final set of regression tests for the vehicle, and they wanted us to do an untethered demonstration flight with 15 meters of translation before the show. The X-Prize people were extremely dismayed at that request, because it was a chance to crash the vehicle before the show, but I thought it was one of the most reasonable things they have ever asked for from us – demonstration instead of analysis or documentation.

 

Joseph loaded up the crane truck with a trailer behind it, taking five of our vehicles to the show: Pixel, Texel, the VDR, last year’s X-Prize Cup vehicle, and the Crayon. Matt rented a large crew van and tailed him most of the way, detouring to pick the rest of us up at the El Paso airport. We were following right behind Joseph when we crossed the border into New Mexico, and ran into our next dramatic moment (hour). The highway patrol were going to force Joseph to leave everything at the crossing station, because he was in violation of various things, principally a little over the maximum driving hours limit due to an earlier stop, but there were also a bunch of company related issues, like Armadillo not having a federal DOT number to employ a commercial driver (even though we aren’t actually employing him, he is a volunteer) and various other things. Joseph recently got his commercial driver’s license and hazmat certification specifically for these operations, but we were unaware of the company side regulations that we also had to follow.

 

So, we were all sitting there wondering what was going to happen. Would Joseph be allowed to come back the next day and get it after a rest period? Could we hire another driver to take it in? Would it have to be offloaded onto another commercial truck? Would we even be able to get the crane truck, a necessary part of our ground equipment, to the show? Russ jokingly piped up with “I saw a movie once where they called the governor and got a police escort away from a highway patrol station.” Someone else said “Call Peter!” (Diamandis) I would probably get the exact chain of events wrong if I tried to relate them, but a half hour later the officer that was still chewing Joseph out got a phone call on his cell phone, and he wound up escorting us to the Las Cruces airport. No flashing lights, though. J He did issue Joseph two citations, but after reviewing the regulations, we are probably going to contest them.

 

On Wednesday, we intended to do a couple hanging test flights to make sure the vehicles came through the transport ok. The X-Prize people had arranged for us to keep the vehicles and truck in the EAA hangar at the airport, but a couple EAA members showed up and kicked us out that afternoon. It turned out that there wasn’t full internal communication about the situation, and we got back in Friday night, but we were outside the next two nights. Then it rained all afternoon and evening, despite forecasts for clear weather.

 

On Thursday, we were required to do the untethered test for AST, but we still wanted to do a hanging test first. It turned out to be a good idea, because we did have a problem with a leak out of the fuel valve packing that needed to be corrected. We also had a problem that we had never seen before: one of our ground support hoses popped off during pressurization. The hoses themselves were transported in a closed box, and the vehicle had dust caps on the connections, but it might still have been some grit on the vehicle side causing a problem. More likely it was just a firm reminder to us to make sure that the (sometimes difficult) connections are completely, positively seated.

 

We were formally given our launch permit, even though it was still contingent on a successful demonstration flight. It was experimental launch permit number 2, with number 1 going to Blue Origin. An unexpected thing happened then: AST got all reasonable. They allowed us to drop the number of sky screen personnel to two along the line towards the crowd, instead of one covering each edge of the flight box (nothing to hit in the other directions…), and they also approved of our proposed vehicle transfer plan that involved wheeling the flight vehicle away from the staging area on a palette jack to the fully loaded crane truck, which they were previously balking at having 50 meters from the crowd (even though it is DOT legal to drive right next to a school bus). Even more fortunately, they allowed me to make two minor code changes over the next two days. They only involved a total of five ascii characters changing, and we had to re-run all of our vehicle regression tests after each change, but it was still more than we had expected them to allow.

 

The AST demo flight required a ground liftoff and a 15 meter translation, the two key things that we had not repeatedly demonstrated in our testing yet. I had initially considered just translating over 15 meters and back, landing on the same pad, but flying it on the simulator it seemed just as easy to go all the way to the destination pad. They didn’t require a specific altitude, so I just took it up to 8 meters off the ground, and started translating. The good news was that it translated just like the simulator, tipping over about 8 degrees to accelerate, and getting up to 5 m/s velocity. The bad news was that at 8m altitude it kicked up so much dust when it left the pad boundaries that it completely obscured the vehicle from my point of view. I pushed it up a bit higher, but it still wasn’t good, so I decided to just fly strictly on the telemetry and put it down right in the middle of the other pad, 100 meters southwest of the takeoff point.

 

It sets down (I could tell by the change in the engine pitch). I kill the engine. The dust clears. Pixel is sitting on its side, five meters or so from the pad, apparently after having rolled down a bit of a mound. We finally figured out that my assumption that the pads were aligned exactly NE / SW, or at least parallel to the 44 degree runway, was incorrect. The actual angle is something like 52 degrees. We were given lat/long coordinates for the pads, but I really needed them in ECEF (earth centered, earth fixed, an orthogonal 3D coordinate system). In hindsight, if I had done one of: survey the pads ourselves, pushed XPC for ECEF coordinates, or dug up software to convert from lat / long to ECEF, we wouldn’t have had a problem. I made an assumption under time pressure, and it wasn’t correct. If they had decided to make the north / south pad the initial one to use, we also wouldn’t have had a problem.

 

It turns out that Armadillos can roll. Pixel suffered no damage, and we were cleared to fly at the event. This was the first ever flight under an experimental permit.

 

That night we spent some time moving our electronics box between the two pads, and found a couple problems. Our integrated velocity drift in translation over 100 meters was more than double what we were seeing in static hovers, and to actually land on the target pad without hitting our soft abort lines would require “good luck” in the drift direction. The way it was set up, rotating the entire flight profile box to fit the real positions would not have been completely trivial, so I proposed that we simply leave the hard-shutdown safety box exactly where it was at 25 meters off the 45 degree route, but expand the soft abort box from 10 meters to 20 meters, giving us enough room to maneuver manually to the actual destination pad. This was not an ideal solution, but it only involved changing four ascii characters in the source code, which I thought might be feasible to get by AST. Thankfully, they agreed.

 

On Friday, the first day of the show, we ran into two separate problems. When we powered everything up in the morning, we were seeing something like 90% packet loss between our ground station and the flight computer. Our first impression was that it was RF interference from something at the show, probably some wireless network, but after scrambling around and trying to get everything shut down to no avail, I tried switching to my backup computer system (we took spares for everything this year), and the problem completely went away. It looks like it was probably a damaged Ethernet cable in the primary system, possibly from something heavy crimping it in the van.

 

Once we had that resolved, we set out to do our first level one flight attempt. We were ahead of time getting out to the pad and through our checklist up to the lox loading point, but something was going wrong. It is normally an eight minute operation to load a small dewar into the vehicle for a 90+ second flight, but after twenty minutes we still weren’t loaded. It turned out that there was ice in the dewar liquid line, almost certainly from having to store it outside in the rain Wednesday night (we had used a different dewar for the previous tests). The judges agreed that being kicked out of the hangar we were promised was a valid excuse, and agreed to reset the clock to the start of lox loading after we switched to another dewar. The rest of the process went perfectly, and we started our first contest flight. I was able to light the engine right on their countdown every flight of the event, which was probably a bit of an accomplishment all by itself.

 

For every flight, I would start the engine, pick the vehicle up a few meters in the air, pause a bit to check that the gimbal trim looked fine, then climb to 55 meters altitude and begin the translation. The up and over went fine on every flight, but once the vehicle was at the point where I had originally thought the pad was going to be, Russ and I would start looking over at a monitor that was receiving the downward facing camera transmission, and try and pilot the vehicle above the actual pad location. This was not easy. The vehicle is intentionally programmed to respond very sluggishly, because my top priority was to not let it get anywhere near the automatic abort / shutdown tilt angles. This means that there is a multi-second lag between the time I mash the joystick over and the time the vehicle actually stabilizes at a new position. Trying to learn how to avoid pilot induced oscillations on a system that has never translated more than ten feet before, in front of a huge crowd, is rather stressful. Russ sat behind me and tried to decide on directions to move, while I was actually holding the joystick and monitoring the telemetry signals.

 

One surprising thing for us was that the distances were not what we thought they would be. The 100 meter translation looked much shorter than we expected, but when the vehicle flew up 50 meters, everyone near the pad was craning their necks back, thinking: “That’s high. Really, really high.” I never saw it, because I was inside the van, just watching the altimeter bar on the telemetry screen rise. We were quite close to the flight path line.

 

We managed to set it down on the pad, but when it landed, all four landing legs broke off. This was a successful qualifying leg for the prize, and I was preparing to fly it back “on bloody stumps”, but we also cooked the drive and feedback cables on one of the gimbal actuators, so we couldn’t make the return flight. It might have been possible to patch the two cables on the pad inside of the half hour or so of cushion we should have, but we didn’t know the exact extent of the damage, and decided it would be better to give it a good checking over that night and try again on Saturday. We borrowed all the legs from Texel to repair Pixel.

 

The target descent rate was 1.5 m/s, but we managed to hit an unfavorable dip in the control system, and actually landed at about 2.1 m/s. The legs clearly need to be stronger, but we asked AST to allow me to change the target descent rate from 1.5 to 1.0 m/s to attempt to soften the landing. They agreed, so we made the change and re-ran all of the regression tests on the vehicle. We had originally thought that the fire damage was probably just due to backsplash since the vehicle was closer to the ground without the legs, but on closer inspection we found that our brand new engine chamber had some serious cracks at the top, and fuel was probably sprayed through the graphite very close to the actuator. We have seen cracks in all of our other chambers, but this looked different, so we believe that what might have happened is that one of the weights on the legs bounced directly underneath the engine, and the engine landed directly on it after the vehicle rebounded. We swapped it out for one of our older chambers. We got to stay in the EAA hangar that night, apparently after the chain of authority was properly worked out (thanks, guys!).

 

On Saturday, we got off to an early start, moved the vehicle out and got it in the air, right by the numbers. Unfortunately, we didn’t quite get the piloting right for landing on the pad, winding up with two legs on and two legs off, letting the vehicle tumble over again. My reaction did not go out on the live telecast. The slower descent rate was noticeable, and we didn’t hurt anything, but since this was not a qualifying first leg, we had to go all the way back to the staging area and start over. One positive thing we were noticing was that with all the extra film cooling, the chambers were barely getting hot after 90 second flights. We didn’t have the presence of mind to actually measure the temperature with our IR thermometer, but they couldn’t have been more than 400 degrees or so, based on splashing water on them (they probably got hotter after the throat heat fully soaked back). We could have lived with less film cooling.

 

We refilled all the consumables on the truck before the next attempt. Helium was the biggest problem, requiring a tedious cascade fill from discrete bottles. They will have a tube trailer next year. We again got out and in the air right on the numbers, and we managed to land it on the pad this time. Just barely. The judges could conceivably have ruled it out of bounds because the spheres overhung the edge a little, but they called it acceptable. The landing broke one shock off. The judges had told us before the flight that we would not be allowed to remove other legs if only some of them broke. I’m not sure what the reasoning was, but I couldn’t feel too put upon by it, since if we had just made the legs good enough in the first place, the issue wouldn’t even be there.

 

We propped the vehicle back up level with a broken leg and strapped the 10 pound weight normally mounted to the leg (part of our required “payload” for the challenge) onto the vehicle with a strap. We were all refueled and ready to make the return trip in less than thirty minutes. This was the last flight window of the show. It would have been so poetic to win the prize on the last chance, just before the show was closed down, flying with a broken leg. The Cup people were breaking out the Champaign. Unfortunately, it did what we though it quite likely could do. When the engine came on, it kicked the propping leg out, causing the vehicle to tip that direction (the opposing shock still had about 200 pounds of spring load). If there were no safety interlocks, the vehicle would probably have done a dramatic swoop around to right itself, but at 15 degrees of tilt angle the flight computer went into abort mode, throttling the engine down to attempt an auto land. It continued to rotate, forcing a complete engine shutdown at 20 degrees. This resulted in Pixel taking a short sideways dive off the pad. We definitely weren’t going to fly again. If we had a couple more hours, we would have pulled Texel out and had a go at it with some kind of cobbled together landing legs (leather wrapped PE foam hose clamped on, probably), but Pixel looked to be down for the count.

 

Sigh.

 

We had two problems – the pad locations, and the landing legs. Both of those could have been corrected if we could just go back to our shop and work for a day or so, but live at the show, under the watchful eye of AST, we had to just do the best we could with what we had. We flew great, we just didn’t land so well.

 

My wife dressed me down pretty hard for mis-managing the attempt since the challenge was finalized. Her position from the beginning was that we should have concentrated exclusively on the level 1 prize, and ignored the level 2 prize. She is probably correct – if we had just taken the injector from the VDR, even with the extremely crummy Isp, and concentrated solely on doing flight tests with that instead of injector development, we would have gotten to the point of doing ground liftoff tests early enough to find out that the legs were inadequate and make a fix. A couple weeks before the event, it was clear that even though we did manage to get all the known technical issues solved, we really needed another month or two of testing to wring out unknown issues and have high confidence that we could pull it off repeatedly.

 

I really wanted to win some prize money, so I would be able to give the team members at least a small bonus, allowing them to get something for their SO to help make up for all the nights and weekends away. The prizes were never make-or-break for us, but we would have accelerated our development if we had some more outside income, and it would have been nice to actually have a profitable year. The Nvidia sponsorship was very welcome, but none of the marketing ideas that were originally discussed came to fruition in time for the event, so I’m not sure if they are able to leverage enough value from it. I tried to get the side of the vehicles with their logos on them in as many photos as I could, but we didn’t have time to produce our own banners and such with a more prominent sponsorship message.

 

I’m not at all sure that holding contests like this as the main event of a show like the X-Prize Cup is a good idea. It came out well this year, but it was all just one mishap in testing away from not having any real meat in the show. If you know for sure that you have a real field of contestants it will probably work out, but if the field is “probably one”, it gets real dicey. My official bet is that there will be no more than one other competitor next year, and it may well just be Armadillo again. Masten is the closest, but they still need to fly their very first test vehicle, then design, build, and test a more potent vehicle to even be able to compete for the level 1 prize. We spent six months and about a quarter million dollars in direct pursuit of this, and we had a running start at it. For many things, time can be traded for money, but there are limits. One hundred thousand dollars cash out of pocket is probably the minimum amount that someone needs to be prepared to spend to be in the game next year, and that would be for a single vehicle, relying on luck to not have any mishaps.

 

I sort of wish I was able to actually “attend” the event, rather than being at the center of it. There was a lot of stuff I would like to have spent some time looking at, and a lot of people I would have liked to talk with. I did at least get to give my respects to Anousheh Ansari as we traded off between interviews with Miles O’Brien. I thought she had done a wonderful job with her blogging from space – I followed it every day she was up there.

 

Almost exclusively, the first word out of everyone’s mouth that I have seen since the event has been “Congratulations!”, which still takes me aback a little, considering we didn’t actually win. We did put on a good show for everyone, and the parameters for this contest do seem to be past some threshold where everyone takes the flight accomplishment seriously. I was told that many of the industry professionals were extremely impressed.

 

On closer examination, Pixel may not be dead after all. The tanks and frame still seem square and tight. We are going to hydrotest everything, and possibly re-weld the computer mounting points and the other things that are broken. I still want to do side-by-side vertical drag racing demos with the pair of vehicles.

 

We are going to take a little break to recuperate and catch up on our other responsibilities, then get back at it to nail all the known issues and proceed on to bigger and better things.

 

John Carmack

10-23-2006