2009-09-14 update:

It has been a long time since I did an update, but it hasn't been because nothing has been going on.  Quite the opposite, in fact -- things have been incredibly busy.  Outside of Armadillo, the big new for me personally was that Id Software, my video game company, was acquired by Zenimax Media.  This doesn't have any real impact on my relative time commitment to Armadillo, since I expect to continue devoting the majority of my time to software for the foreseeable future, but it does mean that I have more personal resources to call on if necessary.

I would have had Phil provide some updates, but he has been overloaded dealing with our FAA/AST waiver applications and other business logistics on top of actual fabrication work.  Our full time head count is going to be going up some soon, so there may be hope for more frequent updates in the future.

This will be the technical update, I'll leave it to Phil and Matt to put together the media and color update.  The X-Prize website has some early videos of the flights, but I expect that there will be some great footage from the NASA TV crew later.

http://thelaunchpad.xprize.org/2009/09/2009-ngllc-videos-of-armadillos.html

The Scorpius Super Mod

Last year we had done several 180+ second flights with Pixel before the Challenge event (as we had the year before that...), but when we tried to fly during our official window we had a valve movement problem and burned through the engine, flipping the vehicle onto its side.  The damage was minor, and when we got back home the vehicle passed a hydro test, so we could have just dusted it off and flown it again, but we has a couple reasons to change direction.   The module vehicles fly so much nicer than the quads -- the control authority is much better, they don't have propellant trim issues, and they have slosh baffles inside the tanks.  We also knew that the new "amateur rocketry regulations" were due to come into force this year, which would allow us to fly in many more areas than the experimental permits we had been getting would, and with the LLC this year allowing you to make your attempt wherever you wanted, it would be a huge advantage to be able to fly at our home base.

The problem with this is that there is a firm limit between what you can fly under waiver and what needs an experimental permit or a full launch license, which we were not going to be able to get anywhere near our hangar.  That limit is 200,000 pound-seconds of rocket impulse.  When Pixel did a 180 second hover flight, it ran  about 240,000 pound-seconds, and there wasn't going to be any way to nickel and dime it down by 20%.  The mod vehicles just couldn't do the mission at full propellant load in our normal blowdown pressurization mode, so we had to add an external high pressure helium tank and fill the tanks completely full instead of just half full to carry enough propellant.  This became the "Super Mod".

The February Armadillo update had a lot of details on the catastrophic results of our first hover test in this configuration.

There wasn't anything deeply wrong with the setup, but it was several months before we got around to trying again, since we were busy with NASA and Rocket Racing League work.  Using a ball valve for programmable high flow pressure regulation works just fine once the kinks were ironed out.  Based on our later experience, we didn't have enough helium loaded to fully pressurize the propellant, so it would have settled back down on the tethers sometime towards the middle of the burn if it hadn't self destructed.  Even if we had the helium, it probably would have run into fuel depletion at about 160 seconds, with a fair amount of lox remaining.

Before we tested again, we went on a weight savings program.  The legs were the most obvious thing -- we had designed them to hold up a bolted together quad-module configuration, so they were much heavier than necessary.  The new legs were smaller, shorter, and thinner, resulting in a savings of 60 pounds, as well as being easier to land among the boulders and craters of the level 2 moonscape.  We also lightened up a number of other things, and there was probably another ten pounds we could have saved if we had to.

The next couple tests were successful, but we found that we couldn't get enough helium for a full 180 second run into the carbon fiber tank we were using with just normal helium tanks that come at 2500 psi or so.  The painful solution was to add some 6000 psi helium tanks to our loading procedure, and cascade fill first from the six-pack of normal bottles, then change over to a regulator on the 6000 psi bottle to bring the vehicle bottle up to 4000 psi or so.  This was a workable solution, but inconvenient and more expensive to operate.  This was the "single black tank configuration".

Unfortunately, it was still slightly over the 200,000 lb-sec waiver limit.  It could make it, just barely, if we offloaded propellant to the point that it would sputter out of propellant at 181 seconds, but then any tiny variance in the process or environment would have it dropping down early.  There were two paths to take for improvement, we could either increase the engine specific impulse or reduce the dry weight.  We pursued both in parallel.

The engine was essentially the same as we used on Pixel, but because the Super Mod was several hundred pounds lighter, the engine ran at a lower throttle level and chamber pressure through the entire flight, which also meant a lower specific impulse.  We started working on another injector that used the same design, but had a greater number of smaller injection elements, and we had some new cast chambers made with 3.25" throats instead of 3.5" throats to increase  the required chamber pressure for hovering.  We didn't get to the point of assembling these, because the weight savings came together ok.

We pulled a little more weight out of the vehicle, but the only quick thing that could save significant amounts of weight would be changing out the high pressure helium bottle.  The bottle we were using was a relatively old Lincoln Composites compressed natural gas tank, and weighed about 60 pounds.  After searching around a bit, we had a conversation with Microcosm about their Pressuremaxx linerless composite tanks.  They were very interested in the project, and offered to become a vehicle sponsor with discounted tanks.  Hence, the "Scorpius Super Mod".

When the tanks arrived, we were very impressed.  They were only 12 pounds each, and had a combined pressure x volume capacity greater than the single 60 pound tank we started with.  Designed for 2200 psi operation, they would also not require the separate 6000 psi helium supply.   This was the "dual black tank configuration" that there were a couple pictures of.   Unfortunately, on our first loading operation, we heard a SNAP sound at 1400 psi and one of the tanks started leaking down.  We weren't going to be able to fly on it, but we continued testing the other tank and it did the same thing at 1800 psi.

We were quite disappointed, and a little worried about the schedule.  We sent the tanks back to Microcosm for analysis, and they agreed to build a second set of tanks as soon as possible.  The final verdict was that two factors contributed to the failures: We had the tanks floating in their mounts with leather pads on both ends, but we had only allowed about 1/8" for expansion, and they believed that the tanks could expand up to 3/4" in length before bursting.  The other factor, which took me a while to accept, was that the tanks had overheated during pressurization.  We were pressurizing at a rate of about 10 psi a second, which was slower than we had been pressurizing the other tanks, but being linerless, there were load bearing fibers and matrix essentially directly exposed to the helium.  They suggested that because we were filling the tanks from the bottom, the helium on the top was getting more and more compressed and heated with minimal circulation to the rest of the tank.  This fit the evidence, in that both failures were near the top of the tank, and when cut apart, there were clear signs that the matrix had gotten very hot near the broken area.

While Microcosm was rushing to build new tanks, we had more tests to do, so we put on four other composite pressure vessels we had laying around.  These were 15 pounds each, so the weight was the same as the single big black tank, but they were designed for operation around 2000 psi, so we could use the same loading procedures we were planning for the Microcosm tanks.  This was the "four green tanks" configuration that we posted a few videos of, and we did back-to-back 180 second flights with it.

While testing of this configuration, I hit on an idea that wound up being very useful -- we set up a counterflow heat exchanger that ran the lox vent through the outside, and the helium fill through the inside.  This let us get significantly more helium in the tanks, and completely quashed any worries about overheating the helium tanks.  They actually start to frost over at the beginning.

The replacement Pressuremaxx tanks arrived a week before our contest window.  We had a Microcosm engineer come out and assist us with the acceptance testing and installation, and everything went fine.  The only change they made was to make them white for better use in the sun, so this is the final "white tank configuration" that we flew at the LLC.  On the vehicle side, we arranged for a full inch of potential length expansion, and filled the tanks from the top to prevent the heating piston effect, although our lox heat exchanger makes that a rather moot point.

We had a legal vehicle, but the propellant depletion was not completely even, so we wound up with something like 35 pounds of fuel remaining at oxidizer depletion.  We spent three more flights tweaking in the mixture ratio so that we would have usable propellant at the end for margin, then one final repeat flight to make sure it was doing what it was supposed to.

When the vehicle was down to its contest weight, the throttle was only at about 50% through the entire flight, rather than the 70% that we were at for most of our earlier long flights.  This makes the Isp rather bad, but it also means that the vehicle could accelerate quite quickly if we were shooting for altitude.  It also caused the hover control code to modulate the engine a lot more than it used to, so I modified the control parameters to smooth it out a bit.  I also wound up tuning the attitude control a bit, since it turned out that our "rocket tug of war" optimized parameters tended to make the vehicle sway around more at hover than it used to.

We had 15 full loads on the super mod before our Lunar Lander Challenge attempt.  One of those loads was scrubbed when the original Microcosm tanks failed, and the first flight ended catastrophically after only a few seconds, but that was still nearly three million pound-seconds of delivered impulse from the vehicle, not even counting the previous flights as a normal module.  That is around $30,000 in direct testing costs.

Lunar Landing

Because we built these pads ourselves, we made sure that they were on a precise east / west line, and exactly 60 meters apart.  We build Pad A sturdy, with good thickness and reinforcement, because we expect to be using it for all of our upcoming boosted hops, replacing the steel plate in a clearing we have used for the recent NASA related free flights.  The lunar Pad B is sort of a throwaway, but we made the boulders removable, so we may fill in the craters later if we need a backup pad.  It cost about $8000 to build the pads and gravel roads out to them.  I am very glad I didn't skimp on the gravel, given the weather we wound up dealing with.

In all of our previous free flights, I had been operating the launch control computer from inside a truck or van quite close to the launch location.  Since we are going to be able to do a lot of flights under waiver at our home base, we have set up a "mission control" inside the air conditioned office space at the hangar.  The launch pad is around 2000' away, but we changed to some patch antennas for communication, and I was actually able to increase our transmission bit rate while improving the quality.

Since I didn't have direct visibility to what is happening on the pad in this configuration, we set up a camera that could see both pads and give me a view on a monitor.  It wound up taking 2500' of Cat5 cable to make the run, and that is the hairy edge of feasible for a composite signal.  None of the modern TVs would acknowledge the signals existence, and even on an older one that would display it, the color was almost completely gone.  Still, it was good enough for me to see the vehicle in the first 30' or so of altitude.

We have used a couple different types of on-board video transmission systems in past years, and they have all had various issues.  This year, I decided to try using IP cameras and the existing telemetry link.  This turned out to be a success, with some caveats.  The focal planes varied quite a bit from unit to unit on the cheap ones we used, and the included software had a lot of issues.  The cameras just send over jpeg frames, and I could set up a combination of resolution, update rate, and compression that I was satisfied with, but the recording feature of the software transcoded the jpeg stream into, of all things, IV5 format, which turned it into utter crap.  This wasn't a huge issue for us, because we also included separate HD cameras on the vehicle, but it frustrates the software engineer in me.  I will probably be integrating direct support for IP cameras in my mission control software so I can automatically log snapshots during the checklist, and record the flight video as a full quality mjpg.

We didn't want to risk the Super Mod in free flight, but we had other flying vehicles on hand to test the traverse and landing.  We started out with the methane module that we had been flying for NASA, first just doing a normal tethered hover over the lunar pad so I could see what it looked like, then moving to free flights from Pad A to Pad B.  We wound up having a burn through on the methane engine on the first pad to pad test.  The methane engines continue to give us trouble when throttled down, and we need to do more development if we want to use them operationally for VTVLs.  The burn through led to the vehicle sliding away from the pad, aborting when it tilted too much, then shutting down and dropping a few meters to the ground.

Aside from mangling a couple legs, It turned out not to have hurt much of anything, so we converted it back to an alcohol fueled rocket and had it flying again that weekend.  One weight savings that we made on the super mod, and also made to the other mod for equality, was to take out the manifold purges and their plumbing.  In hindsight, this wasn't worth the savings.  The engine starts and runs fine, but it doesn't shut down cleanly, leaving licks of fire going for a while.  I can sort of pump it out by using the igniter gox solenoid, but it is a mess.  This turned out to be more of an issue with ground touchdowns than for tethered tests, because there is a little puddle of burning alcohol close to the vehicle.  I didn't want to touch the perfectly working Super Mod at that point, but we are going to put them back on this week.

My first landing was a few feet off, and I actually bashed one of the boulders with a foot, but it didn't upset the vehicle much.  We spent more time orienting the cameras so that they were exactly straight down, and the second landing was less than a foot off the center mark.  We will probably go to cameras that aren't quite so cheap in the future, it was frustrating that the optics weren't aligned very well with the case.

The Challenge

Our biggest worry was the weather.  A week out, it was expected to be raining all weekend, and the forecast never improved.  We had a local television crew out on Thursday night to witness what we expected to be our last flight test before the challenge, but the rain rolled in just as we were preparing for it.  We did manage to get that final test done on Friday morning, and when the judges and X-Prize representative arrived that evening, I was semi-seriously suggesting that we just go do the flights right then, instead of waiting for Saturday.  It wouldn't have been valid without the NASA representative on site, and Peter Diamandis would never have forgiven me, but it was a very tempting notion.

Saturday morning, it was raining.  Lots of people had weather displays going on phones and laptops, and it looked like there was a clear section that should pass over us, but it wasn't moving.  The judges and X-Prize officials had all been as reasonable as we could ask for when we discussed contingencies.  If we aborted due to weather before the first flight, we could start over.  If we aborted after one leg was completed, we could pick up from that point later.  If it rained all weekend, the judges and X-Prize official would be willing to stay through Monday, although we would also need to convince a NASA rep to do the same.

After surveying the pads in the afternoon, we were very worried about getting our trailer stuck behind the crane truck, so we decided to use a smaller trailer towed by a separate vehicle for the rocket, allowing the crane truck to go straight in and out to minimize the chance of getting bogged down.  We discussed various contingencies, including having me fly the rocket over the crane truck if it got stuck by the lunar pad.  We would have had to offload residuals into a carboy and carry the rocket back, but we could have managed.

Finally, at 3:00, it stopped raining and we rolled out.  It did start misting during our loading operations, but we felt it was reasonable to keep going.  We got the fuel and lox loaded, but when we were nearing completion of the helium fill, Russ noticed that there was a leak from the high pressure servo valve packing.  Fifteen previous flights had never seen one.  This was tense, because not only was the contest timer ticking, but our lox was also boiling off at about three pounds a minute, so if we had a 15 minute delay, I would be very worried about running out of lox while over the pad.

TODO: valve stem leakage is a chronic problem, we need a better torque spec and locking mechanism.

 The pad crew got it done in about ten minutes, and we were ready for liftoff.  I triggered the ignition.... and got an igniter misfire.  I realized that I had skipped the checklist step to bleed the gox line to the igniter.  I was using the same "checklist_superMod.txt" file that we had been evolving through all of our testing, but because this was an untethered free flight, there were several steps that would be skipped around ignition time, and in the excitement, I skipped one that I shouldn't have.

TODO: create event-specific checklists.

TODO: modify the ignition timings to gox lead automatically and / or change to the more reliable igniter geometry on the rocket racer engine.

After I got the engine lit, it lifted off nicely.  I had decided to burn off the first 20 seconds of propellant close to the ground at the launch point, because that was the part that was hardest on the engine, and I wanted to be able to get it down safely if it had a burn through.  After running that out, I ascended to 55 meters, moved over the 60 meters to the second pad, then descended to 10 meters to run out most of the clock.  With 40 seconds to go, I dropped the altitude to only 3 meters for the final precision dial in.  As I was trying to make the final adjustments, it didn't seem to be responding correctly.  It landed with one foot in the center of the pad, or about three feet off center.  It was also rolled 90 degrees off from where it should have been, which shouldn't have been possible, since the roll control keeps it within +/-10 degrees.

When the pad crew got back to the vehicle and we started our detank procedure, we found that the lox hose couldn't reach the vehicle at the current orientation, and we didn't want to move the crane truck off the gravel road.  We couldn't move the vehicle until the judges noted the vehicle position, and the judges couldn't approach until we removed all the propellant from the vehicle.  We decided to just dump the lox through the engine, which was also faster.  This worked fine, and the fuel hose was long enough to reach the truck, but it got us off our checklist, which had a later effect.

I had noticed a couple things out of the corner of my eye when I was doing the flight:  the telemetry had some bad spots, and the vehicle shadow sometimes looked like it was at a 45 degree angle, which wasn't normal, but we were in higher winds than our previous flights, so I hadn't worried too much about it.  If I had been aware that I wasn't at the roll angle I thought i was, I probably would have been able to figure out the right way to nudge the vehicle for a perfect landing.  As it was, it would have been better if I hadn't even looked at the on-board camera views.

TODO: draw a heading arrow on the vehicle shadow in my flight display.

TODO: multi-person mission control with additional displays, so we can have another set of eyeballs watching data.

This was extremely tense for me.  I reviewed the telemetry and found that it had fired one pair of roll thrusters early in the flight, but that had caused it to slowly sail past the point where the other pair should have fired, and it just kept going.  It didn't even try to fire them.  I knew what was the probable cause for this, but it didn't initially make sense.  We had known for a while that one driver channel on this electronics box was flaky, working sometimes and not working others.  This only controlled one if the redundant roll thrusters, so it wasn't a big deal, but one of the last things we did before the contest was to move that pair of roll thrusters to a different driver channel.  Each connector is either a bidirectional motor or a pair of solenoids, so moving that one connector moved a pair of roll thrusters.  We did a full 180 second test flight after this change, and everything worked fine.  When I looked back at that test data, what I found was that the roll thrusters hadn't fired during that flight either, but it had still stayed perfectly straight all by itself.

We test every actuator before each flight, and the roll thrusters worked perfectly when I manually fired the bits that they were on.  The problem was that the patch to the code to remap the channel was wrong.  I made what I thought was the correct fix, and we started the checklist for the second leg of the flight.

TODO: create command-roll-left and command-roll-right buttons that can force computer initiated testing the roll control under tether.

As we started the second leg, we got bit by the circumstances at the end of the first flight.  When we had to improvise the offload procedure, we missed the step in the final checklist where we vent the residual  pressure in the helium tank.  In the aforementioned actuator test, I cycle the helium regulator servo valve open and closed.  The way we added the external pressure tanks to the existing mod system was by having a couple hoses from the servo regulator that could be plugged into the existing pressure / vent valves.  These hoses are free hanging during the early part of the loading operation, so when I tested the regulator valve with pressure still in the tank, they flailed around and hit Russ and Mike in the head.  This is the worst safety incident we have had in almost ten years at Armadillo, and is a great example of why you should wear safety goggles during the entire process (which is checked on our checklist).   They both got a good lump from this.  I didn't even find out about it until after the flight when I asked Russ why he had a pad duct taped to his cheek.

TODO: add an explicit "verify high pressure vent valve is open" step before the actuator test.  We have this for the main propellant tanks.

TODO: modify the gas manifolds so they have a dedicated, permanent port for the regulator gas, instead of a manual connection.

As we started going through the checklist for the second flight leg, I noticed that the pad camera television and the on board camera display computer were both shut off.  We had had a power outage at the hangar.  It hadn't effected the laptop that I run the main flight control program on, but it must have also interrupted the wireless radio that communicated with the rocket.  If that had happened during the flight, the vehicle would have gone into an auto-land from wherever it was.

TODO: get a UPS for everything at mission control.

When we got through loading everything, I made damn sure that I bled the igniter gox line this time, but I still got an igniter misfire.  I tried again with the same result.  The fuel tank pressure had also oddly dropped a lot.  I realized that I might have made a mistake in my panicky fix for the roll thrusters, and when the pad crew mentioned that they had heard a roll thruster fire when I tried to start the engine, it was clear that I had moved a bit the wrong way.

TODO: take more than 60 seconds to evaluate changes to flight control software.  When possible...

One more build and we were off.

I was prepared to try to manually keep the roll in bounds by occasionally pressing the actuator bit that fired a thruster if it looked like it wasn't working, but this flight was completely nominal, and the terminal guidance also worked properly, so I landed one foot from the center.  The offload procedure was also completely nominal.  We had a half hour to spare.

While the judges were verifying the position, the pad camera display started flashing "low battery", and went dead a few minutes later.  That wasn't critical to any of the operations, but I would have been distressed if it had happened while I was flying.

TODO: check all camera charge levels before important operations.

It was great to win in front of the home crowd.  It is bad to fail, but it is worse to fail in front of your wife...

Concluding Thoughts

The home field turned out to not play a part in the success this time.  I was sort of expecting some mad dash back to the shop to build something for a second try, but we didn't need it.  Back in 2007 at our first try, it would have been the difference between losing and winning the level 1, since we would have been able to just cobble together some rubber bumpers to replace the unusable shock feet.

The challenge would have been better if it didn't include the loading and unloading of the vehicles from the transport vehicles.  It was silly to have the judges declare our flight valid, and still have to complete the loading of the vehicle back onto the trailer.  People should be justified in cheering when the vehicle lands.

If NASA wants to do a Level 3 Lunar Lander Challenge, my suggestion would be to have it be a single system that makes a 120 second flight to a lunar pad, has a short stay time, then makes a 120 second flight back to the launch pad.  Contestants choice to either stage for the flight back, or have a very high performance vehicle do it in a single stage.  This would be much harder than the current level 2 challenge, and we could not do it by cobbling together a vehicle from the things we have on hand.  It isn't clear that Armadillo would be participating, because we have enough business prospects that we may be too busy satisfying customers to compete.

Checklists are very, very important.  It is tempting to just assume you know what you are doing, but experts still make plenty of mistakes.  On the other hand, it is important to be able to improvise when reality throws you a curve.  There have been some interesting articles in recent years about the value of, and resistance to, checklists in the medical fields.  I am quite proud of the checklist integration in my control software, but we should continue to improve it with integrated note taking and audio / video logging.

It is clear that we can add significant performance to a blowdown vehicle design with external pressurization, and this was a useful experience, but a blowdown vehicle is faster, easier, and cheaper to operate.  We probably won't operate an externally pressurized vehicle again in the next year.  There are too many things that we need to do that can be done just fine with blowdown.  However, It is very likely that Microcosm tanks will show up  in a lot of our design studies in the coming year.

 If we did wind up doing a lot of work with high pressure helium, we would want to invest in a high pressure pump so we could transfer helium around from the partially emptied six-packs that you are left with after a cascade fill.  If we started using that, it might also be tempting to filter and reclaim helium from the propellant tanks, but I don't think the hassle is justified yet.

Coming Up: higher altitude boosted hops, scientific payloads, more rocket racer flights, and multi-module configurations.