September 4, 2007 notes:
Texel Crash
(note that this is slightly edited
from the original posting to aRocket with a few more
bits of information)
August 18 was a bad weekend for Armadillo. We set out to put some flights
on Texel, the backup Quad vehicle, and it
didn't go so well.
We started out with a normal 90 second elevated / tethered hover test, but we
ran into a problem with the actuator power. We initially thought it was a
bad main power switch, but it turned out to be the lithium-polymer battery pack
cutoff circuit incorrectly shutting down at 16 amps of load instead of
40. This was a new battery pack ( www.batteryspace.com HPL-8059156-4S-WR), and it had passed all the
individual actuator checks, but when the igniter started firing with both high
amp NOS solenoids, the battery shut down (went to 0.3 volts indicated) after
one second and stayed there until it was physically disconnected. Russ made
a fairly heroic field repair, cutting open the battery pack and wiring around
the protection circuit while sitting on top of the rocket. The total time
spent on this after three attempts was 90 minutes, and enough lox had boiled
off that the vehicle hit lox depletion at 60 seconds of flight. We got a
few good data points from this: the batteries need to be checked at full
current load, with vents open we boil off about two pounds of lox a minute, and
lox-depletion runs are benign, if a little flamey.
For the second flight we were going to do a ground liftoff (still tethered for
runaway protection) to test the automatic ground contact engine shutoff
code. We have had several reasons to want to automate this: We get a fair
bit of bounce on touchdown, because the engine is essentially keeping the
vehicle weightless during the terminal descent. A computer controlled
shutdown would be at least a half second faster than my manual punching of the
shutdown when I visually see ground contact. The quads will just safely
bounce around on the ground a bit if the engine just goes to idle and doesn't
shut down, but the module, with the gimbal below the
CG, will try to tip itself over when a landing leg becomes a pivot point, so
there is extra incentive to get it shut off fast. You can see that in our
XPC '05 vehicle flight. We also need to handle the case of the vehicle
landing in a situation where I can't shut the engine off promptly, either
because there was a telemetry problem, or when we are doing high altitude
flights, it lands out of direct sight. There is a separate shutdownTime parameter that will keep it from sitting there
at idle for ten minutes, but a telemetry abort could still have it on the
ground and cooking for the better part of 220 seconds. We could still
shut the flight safety fuel valve, which would result in just idle level lox
pouring out of the engine, but that has its own problems.
I have been very hesitant to put in ground contact shutoff code, because
shutting the engine down for some incorrect reason would be catastrophic, and I
would feel awful if that ever happened. We had some switch based ground
contact sensors on the old VDR, but they never got tested. We have
concluded that the landing jolt, as seen by the IMU accelerometers, is a good
enough ground contact signal. There is always the worry that combustion
instability, or a nozzle ejection event, might trigger the signal level, so
there are additional guards about it only functioning when you are within three
meters of the ground (we must leave some slop for uneven terrain or GPS innacuracy) and trying to descend.
We loaded up again, being very thankful that we now pack three six-packs of
helium for each test trip after we were forced to cancel the second flight on a
previous test session due to insufficient helium after troubleshooting a
problem forced a repressurization on the first
flight. Liftoff and hover was fine, and at the 45 second mark (no sense
pushing it on a ground liftoff), I had it come in for a landing. It hit
the ground, and I saw it bounce back up. My first thought was "That
didn't seem to help at all". My second thought was "Uh, that looks like it is accelerating upwards, not
bouncing." My third thought was "How the heck did the ground
contact code cause that?" My fourth thought was "Crap, its
going to fly into the crane, I need to kill it".
After I terminated thrust, the vehicle coasted to an apogee of about 20 feet,
and fell to the concrete. It made a fireball that would make any Hollywood movie proud, but the vehicle didn't launch
itself back off the ground, make an earth-shattering kaboom,
or throw any shrapnel. The fire truck moved into range of the crash and
hosed down the vehicle until the fire was extinguished. Surprisingly, the
flight computer was continuing to operate and transmit through all of this, but
the sensor and wiring harnesses were shorted out in the fire, so we didn't have
any sense of the state of the tanks. With trepidation, someone approached
the vehicle and found that the lox tanks were still full and pressurized at the
same pressure as when the vehicle was shut down. The Aspen Aerogel insulation on the tanks had prevented them from
even warming up. We vented the lox, and started assessing everything.
It didn't take long to find out exactly what had happened.
On touchdown, the ground contact logic failed to activate at all. The IMU
in Pixel is an older model Crossbow that was rated for +/-10 Gs, but reads to
+/-14 Gs. That particular model was discontinued, and the newer IMU in Texel was only rated for +/-4 Gs. I had set the
ground contact trigger value to 5 Gs, which I had some recollection that the
IMU read to, but it turns out that it was maxed out at 4.5Gs.
What caused the upwards flight was a GPS issue. On ground contact, the
GPS PDOP value went from our normal 200 or so up to a value of 1200. The
very next frame, it went to 0. We ignore GPS updates when the PDOP is 0,
and also some other cases where we know the data is bad, but after flight
starts the rule has been that any valid GPS update is taken as authoritative
for velocity. Between GPS updates, and if the GPS goes invalid, the IMU
will use dead-reckoning to coast for a while, but the accelerometers in
particular aren't accurate enough to do this for very long. The at-impact
1200 PDOP update contained velocity values that were significantly off,
including a 5 m/s down velocity, which caused the vehicle to throttle up to try
to regain the desired 1.5 m/s terminal landing velocity.
I briefly wondered if the GPS antenna mast might have actually fallen off on
landing, giving a correct velocity before losing sat view, but reviewing the
video showed that it clearly stayed in place through the bounce.
We have known these GPS receivers are vibration sensitive for a while, and we
take several measures to protect them from the rocket environment, but this was
the first time we had a shock related failure. I went back to the
telemetry from the Oklahoma
free flights to look for similar signs on touchdown, and found a corroborating
point. On the second free-flight, the GPS PDOP jumped from 200 to 359 at
the time of touchdown, as seen by the accelerometers. The first ground
touchdown of the module also showed a jump in PDOP at touchdown. This effect has evidently always been there,
and some combination of the still heavy propellant tanks from the shortened
flight and just bad luck caused a jump all the way to 1200 PDOP and an unusable
value.
The question of interpreting GPS PDOP values has always been an issue for
us. The exact meaning has to do with uncertainties in the calculated GPS
position value due to the angles between the currently tracked sats. Flights typically have a PDOP between 150 and
250. We have a no-go set at PDOP 300, and an in-flight abort set at PDOP
400. While the calculated GPS position can vary widely with higher PDOPs, the velocity value always seemed to stay fairly
reasonable at high PDOP values, so I had intentionally elected to continue
using velocity updates even if the PDOP was past the abort point. The
situation I was worried about was having the vehicle at 60 meters altitude, and
having the PDOP change to 450, forcing both an abort, and, if I couldn't use
that velocity data any more, a reversion to coasting on the IMU updates all the
way to the ground, which I wasn't very confident of. I'm not beating
myself up about this original decision.
The change I have made as a result of this is to define an unacceptable PDOP
value, initially set at 500, beyond which a GPS update will simply be ignored
as if the PDOP were 0. If we are in a situation of degrading PDOP somehow,
that should allow it to abort at 400 and still hopefully make it to the ground
with GPS based velocity updates.
In hindsight, there are a couple other, more reasonable-to-expect, things that
would have saved the day:
If we had tested the ground contact logic by actually dropping Texel at the shop, we would have found that it didn't
trigger, and I would have adjusted the value until it did work. If that
had been done, we might not have even noticed the GPS problem, because it would
have shut down before any action based on the bad velocities was taken.
If I had planned on just shutting down the flight like I normally do, instead
of just watching what the vehicle did with just the ground contact logic, it
would have just looked like a higher-than-normal bounce, and we would have seen
both the ground contact failure and the GPS issue when I looked at that part of
the telemetry. If the ground contact logic had functioned, my pressing
the button shortly after would have had no effect.
There were a number of data points learned from the experience:
The vehicle fell straight down after the failure, as it was supposed to
do. In fact, three separate shutdown systems were activated. I
commanded the shutdown first, followed shortly thereafter by the computer
thinking it had flown outside the shutdown box due to the abnormal velocity and
triggering another shutdown code, followed a second later by Russ hitting the
remote flight termination button and causing the independent flight cutoff
valve to close. The reaction times on all of this were quite good,
especially considering the completely unexpected nature of the failure.
It is one thing to be watching a specific line and hit a button when something
crosses the line, and another thing to analyze a completely unexpected
situation and make a high-cost decision under pressure. Everyone else
reported that their first thought was "Why is John doing a
touch-and-go?", and it took a little while to register that this was not
intentional.
The vehicle had over 200 psi in the tanks when it
went down. There were two pressure vessel failures, both on the fuel
side: one of the stubby "feet" on the bottom of one of the fuel tanks
hit hard enough to tear at the weld, and the fuel pipe that supports the cutoff
valve broke under the force of the impact. All of the fuel was pushed out
in very short order, and ignited by residual flames from the engine
shutdown. No lox was vented, although there were high heat signs of a
small oxygen leak around our lox dump valve during the fire, probably due to
the fire cooking the valve packing. It would have been more dangerous if
the lox tanks had also broken, but it is hard to say exactly how much. It
might have been just an extremely hot fire that melted the vehicle to slag, or
it might have been a significant explosion.
All three remaining tanks were bent in somewhat at the bottom where the
"feet" were, but they still hold pressure. We will probably do
some fatigue cycle tests on them now that the vehicle is scrap. If this
had been over dirt, or the tank bottoms had just been simple hemispheres with
rubber bumpers without the threaded mounts protruding, the tankage
would have most likely been completely undamaged after a 20' fall. The
top fuel pipe would still have broken, but that could be designed around if
desired. It is possible to make a pressure fed vehicle that can survive a
20' drop.
The fireball and blaze were very spectacular, but the parts of the wiring
harness that were wrapped in leather actually came through the blaze ok.
With some more care, it would not be unreasonable to engineer a wiring harness
that could actually live long enough inside a blazing wreck to allow you to do
something useful, like vent pressure.
One point that was brought up afterwards that I hadn’t considered was
that alcohol fires are noticeably easier to fight than kerosene fires, since
water mixes and dilutes the fuel, rather than just spattering it around. Several times a year, I debate with myself
over the benefits of switching to kerosene, and this is one more point in favor
of alcohol.
Everything inside the electronics box appears to be unscathed and still
functional, although we know from a previous crash years
ago that hidden faults may still lurk, so we aren't going to use them
again. We can give an extremely strong endorsement to McMaster-Carr "
85925K423 Fire-Retardant Silicone Foam Rubber Sheet Adhesive Back,
1/4" Thick". We started covering our electronics boxes in foam
years ago to give them a milder acoustic environment for the GPS units, but we
managed to set the electronics box on fire a couple times when test stand
engines misbehaved. We eventually moved to this material, and it not only
didn't burn inside this fireball, but it protected everything inside the box
quite well. The foam on the bottom of the box was hit by a high-pressure
burning fuel jet from the plumbing rupture, but it didn't burn through.
We still have Pixel and Module 1 in flyable shape at the shop, so this doesn't
have a critical impact on us, but it does change our testing plans for the next
two months before the X-Prize Cup. We are cancelling the untethered 180 second flights for Pixel at OKSP. We
will plan on doing two sets of back-to-back 180 flights under tether, but if we
are going to risk a crash, it might as well be for the money at XPC now that we
don't have a backup. We are going to finish up Module 2 in the next
couple weeks so we have a backup for level 1. Modules 3 through 5 should
also be at least frame constructed by XPC, but whether we get them wired and
tested will depend on how our flight testing goes. If we manage to
destroy a module in the next two months, we can crunch hard and get an extra
one put together if necessary.
http://media.armadilloaerospace.com/2007_09_04/texelCrash.mpg
http://media.armadilloaerospace.com/2007_09_04/fixingBattery.jpg
http://media.armadilloaerospace.com/2007_09_04/rupturePoint.jpg
http://media.armadilloaerospace.com/2007_09_04/crash1.jpg
http://media.armadilloaerospace.com/2007_09_04/crash2.jpg
http://media.armadilloaerospace.com/2007_09_04/crash3.jpg
http://media.armadilloaerospace.com/2007_09_04/crash4.jpg
Back-to-back 180s
Seven days after the crash, we did back-to-back 180 second
flights with Pixel, fully demonstrating the performance and turnaround time for
the level 2 lunar lander challenge.
The last 180 second flight we did showed some erosion on the
fuel manifold below the nozzle exit, now that we aren’t hanging a crack-prone
section of the graphite chamber down below it to protect it. For this flight, I made some phenolic rings and a stainless retaining plate to fill that
area. We also increased the diameter of
the four igniter holes to lower the igniter pressure and hopefully get less
erosion.
Matt put parts of this in fast-forward, since watching Pixel
sit there for six minutes is rather boring:
http://media.armadilloaerospace.com/2007_09_04/double180.mpg
The phenolic spacers got burned
through and ejected fairly early in the first flight, and the igniter still did
erode a bit, but we went ahead and flew it again. After the second flight two holes had been burned
in the fuel manifold, likely because it had already been pre-heated by the
first run. This turns out to be
non-catastrophic, because fuel spraying out cools the area, but the increased
fuel consumption caused us to be running on fumes at 180 seconds on the second
flight. There was no erosion after the
first flight, so even though the phenolic rings got
spit out fairly early, they did provide some protection. We are exploring other materials for the
insulating ring, and also different graphite chamber designs that could avoid
the need entirely, while still keeping the graphite in compression everywhere. The igniter holes didn’t seem to erode any
more on the second run, so we can probably just increase the initial size to
keep them from melting at all. The graphite
chamber has no erosion after six minutes of flight.
Pixel had more rocket powered flight time that weekend than
Space Ship One had in all of its flights combined. We have also spent more on operational consumables
(helium, lox, alcohol, truck rental) than the vehicle itself cost, which is
probably a first for any rocket vehicle.
Long Module Flight
Module development has stepped up in importance now since Texel’s drash. James has finished two more module tank sets:
http://media.armadilloaerospace.com/2007_09_04/twoModules.jpg
http://media.armadilloaerospace.com/2007_09_04/moduleConnection.jpg
The remaining modules will probably be done this month. We will probably fly a two-module dual-gimballed configuration after XPC as the first demo of “bolt
together modules”, followed by a four-module differentially throttled
configuration. The two-module
configuration will need extra brace rods on the outside, but the four module
configuration will be very strong with just the tank-to-tank connections.
For at least a year I had been telling myself to track down
the shock stickers that they use on the Mythbusters
TV show, but the Texel crash finally made me
do it. It turns out that McMaster-Carr
carries them…
http://media.armadilloaerospace.com/2007_09_04/shockStickers.jpg
For my birthday this year, my wife got me a crane truck. J
http://media.armadilloaerospace.com/2007_09_04/ourCraneTruck.jpg
In hindsight, we should have gotten one last year. We have paid quite a bit in rental fees over
the last few years, especially now that we are flight testing practically every
weekend. The great thing about owning
the truck is that we can customize it for our loads, which will save hours on each
testing day. All the dewars will get bolted to the bed, the tool boxes
will be hung below the bed, etc. We are
making a roll-up weld curtain to protect the undercarriage, replacing the big
boards we used to set up (hot rocks have a tendency to cut air lines under the
truck):
http://media.armadilloaerospace.com/2007_09_04/weldCurtains.jpg
We are intentionally dragging the performance of the module
down to limit the hazard area at the X-Prize Cup. We are carrying 90 pounds of payload instead
of the required 55, and we are using a smaller engine throat. In addition to having less injector elements,
the injector igniter was changed from four angled holes to just two larger
ones, hoping that it will allow more heat transfer away from the center of the
igniter so it doesn’t erode. We machined
a single graphite ring to replace the stack of phenolic
rings as the insulator for the fuel manifold.
The first flight last week was probably the smoothest hover
we have ever had, potentially due to using the IMU as a leveling inclinometer on
the modules during gimbal calibration.
http://media.armadilloaerospace.com/2007_09_04/module100.mpg
The chamber is perfect, and the igniter didn’t erode at all. The graphite insulating ring was not eroded
at all, and still held firmly in place.
Everything looked flawless.
For the second flight, we set up for a ground liftoff to
check the code modifications made since the Texel
crash. We were rather surprised when the
top of the engine basically popped off.
It looks like some propellant mixed and burned inside the helicoflex seal at the top of the engine. We had seen one case previously where the
jacket had apparently unwrapped itself.
It is possible that the faster turn-around times on the module
contributed to a worse effect on the second flight here. The seals can be made with the open gap
either inside or outside, but the usual practice for the manufacturer was to
put the open side away from the media being sealed, so we put the solid side
towards the higher pressure fuel jacket.
This looks like it was a mistake, so we are going to get some made with
it flipped the other way. We are also
going to try normal viton o-rings again with a minor
change to give it a little more heat protection.
http://media.armadilloaerospace.com/2007_09_04/poppedTop.jpg
http://media.armadilloaerospace.com/2007_09_04/deadHelicoflex.jpg
News
