Lunar Lander Challenge ’08 –
We win one!
We finally won the level one Northrop Grumman Lunar Lander
Challenge. We didn’t get the level two
flights off, but we aren’t too upset about it.
It was a great week! As with the last
two years, Matt will be getting together a nice update with lots of media from
the effort, but you get the narrative update early.
The official X-Prize site has a good highlights video:
http://space.xprize.org/lunar-lander-challenge
The webcast was apparently very well received, and I did a
very long interview after the flights that I hope is archived somewhere.
Preparation
With all the work for the Rocket Racing League and the
methane engine development work for NASA, we hadn’t flown a VTVL vehicle since
January. We figured that with the more
reliable film cooled engines, there really wasn’t all that much that we needed
to do, since the only thing that kept us from winning something in ’07 was
engine start problems. After the bulk of
the Rocket Racing work died down, we expected to spend six or eight weeks
before the contest doing various test flights, especially with Pixel for the
level two prize.
On August 4th, we got a surprise notification
from FAA/AST that tethered vehicle tests are now declared “not exempt from
licensing / permitting requirements”. We
have been doing these tests for eight years, often with FAA/AST people on site
as witnesses. This ruling was a shock,
and while we have several hearsay accounts of the events that led to this
inside the agency, we still don’t know all the internal machinations that led
to this result.
In early October, when we were still at a dead stop, I even
went so far as to write someone in the government outside the FAA that thinks
well of us, and ask for some advice.
This is a pretty good summary of the situation:
I
would like to ask you for some advice.
A few months ago, some busybody lawyer at the FAA-AST office of legal council
issued a ruling that tethered vehicle tests are "launches" and need a
full experimental permit or launch license to conduct. We have been
conducting these tests for eight years, often with AST personnel in
attendance. We acknowledge the necessity of licensing for significant
free flights that go through controlled airspace, but testing a control system
on a vehicle by hovering it ten feet off the ground while it is attached to a
50,000 pound crane truck is just not a "launch".
This is a launch:
http://www.spacex.com/multimedia/videos.php?id=30&cat=recent
This is not a launch:
http://media.armadilloaerospace.com/2007_09_04/double180.mpg
The current interpretation is that they are the same.
The technical reviewers at FAA-AST claim that they have been fighting against
this ruling for the last couple years, but they lost. I'm not really sure
how this works -- the office of legal council apparently trumps George Neild, the actual head of AST on these matters. We
have been working hard to move AST towards using real operational data as the
basis for granting licenses over abstract evaluation, because that is a much
more realistic way to ensure public safety. This makes it a catch-22,
since we can't collect real data without a license/waiver/permit, so the
license/waiver/permit process must rely exclusively on theoretical
analysis. Don't get me started on the issues and limitations of that.
The AST evaluation teams realize that this is basically screwing us over, and
they promised to fast track a waiver for the requirement of a launch license
for our testing. We did the whole waiver application dance for them, and
they did wind up issuing us a waiver with only a handful of itemized
restrictions after only a couple weeks, which does qualify as them doing their
best to help us out. However, there were unintended consequences.
Because our testing operations are now considered "launches", the
permitting of which is considered a "major federal action", the
parent/sister FAA organization has now decided that everything needs to be
given the full treatment. Since we are now "flying rockets"
(even though they are tied to the ground) instead of just performing industrial
testing, various FAA waivers are required, and none of these people feel any of
the guilt that the AST division feels for inflicting this on us. In fact,
there seems to be something going on where the FAA people are irritated at the
AST people about some internal notification issue, and some people in the FAA
have decided to require us to do several things that have the local airport
managers scratching their heads -- like requiring "obstruction evaluation
reports" for us to operate our crane truck on airport grounds.
The NASA lunar lander challenge is in three weeks,
and we haven't been able to test any of our vehicles. Not at our home
base at the Caddo Mills airport, nor at any of our other testing sites at
Hensley Field, the Greyson County
airport, or the Oklahoma Spaceport. Our NASA development contract is also
stalled waiting for a tethered test. I still hope that our FAA waivers
are going to come through any day now, but this has left me deeply frustrated.
This was a bad regulatory overreach, and many of us in the industry worry that
they have their sights set on static engine testing next. We have been
protesting this through the Personal Spaceflight Federation, our industry trade
group, but there doesn't seem to be any traction. We had a great
relationship with Patti Grace Smith, the former head of FAA, but the new
director comes from different circles, and while he has heard our case, there
hasn't been any movement.
As an engineer, I have tried to avoid politics my entire life, and I am
ignorant of all the details. I don't expect that there is any political
route that can help in the current crisis, but I would like to do whatever is
possible to try and get this bad ruling overturned for the future. I'm
sure we will eventually get all of the newly required permits and waivers, and
some could look at that as an additional barrier to entry for competitors, but
it just isn't right, and there are concerns about future steps that may follow.
If you can offer any advice, or possibly an introduction to the correct person
to plead our case to, I would appreciate it.
John Carmack
As far as we can tell, lots of people inside AST, including
the administrator, are really and truly on our side. We don’t see eye to eye on a lot of issues,
but I can usually muster some sympathy for their positions. However, based on this action, there are
people there that I would classify as enemies.
Not “out to get Armadillo”, but definitely taking active steps to make
it harder for everyone in the industry to accomplish our goals. This didn’t happen out of passivity, someone
decided they were going to make an issue out of this and lobby for it. The nearest I have heard anyone come to
trying to defend the ruling goes something like “Any sensible tether design
will be approved by AST and a waiver will be issued, this is just to prevent
dangerously unsafe systems from being used”.
This is the common defense of many laws -- “A just man has nothing to
fear from this law. You are a just man,
aren’t you?”
Whenever I bitch and moan about regulations, I do feel it is
necessary to caveat the statements so people don’t get a mistaken impression of
the relative costs. Some people like to
take a stand along the lines of “The government is keeping private companies
from getting into space! Those evil
bastards!” or the more personal “My company would have
been flying in space now if not for the government!” The former is just not true. The regulatory burden does exist, and it does
cost us – Neil rarely turns a wrench nowadays, spending all of his Armadillo
time doing paperwork, and Phil has been halfway there as well during this
latest crisis. Still, it is only about 25% of the team effort, and it would be
a smaller amount for a larger team. If
you aren’t capable of getting through the regulatory work, it is a damn good
bet that you aren’t capable of building a functioning rocket either. I do still hear some old timers hint about
how AMROC was actively sabotaged, but usually when companies complain about the
government it is just code for “They didn’t give me $100 million to build my
rocket”.
We did finally get all the necessary waivers, and we started
our tethered testing on Ocrober 17th. Hearsay has it that there was an intervention
on our behalf from very high up in AST to get the FAA side of things cleared
up. I made the following post to aRocket as the vehicle was being packed up for the trip out
to Las Cruces:
We have burned well over 10,000 pounds of propellant in the week and a
half since we finally got our tether testing waivers sorted out. We
were very fortunate to only have a couple days of rain, or things might
not have been possible, and I would have been very, VERY bitter at AST
over the tether ruling. As it is, we crammed most of the testing we
wanted to do into one quarter the time we wanted to do it in. We did
lots of full duration burns for both vehicles (Matt has a great video of
another 180 second Pixel flight with the sun literally going down behind
it), a ground liftoff and touchdown, and a bunch of trim adjustments on
Pixel. The only thing we haven't done this year is a full, untethered,
pad-to-pad free flight, although both vehicles have done several of them
in past years.
There are still a few things that aren't perfect:
Pixel has never started a significant translation with a full propellant
load. Several flights have been made at XPC '06 and in Oklahoma, but
they were all with half propellant load for 90 second flights. We could
choose to burn off the first 90 seconds of propellant over the starting
pad before doing the traverse, but that would have us in full slosh
wiggle during the motions, and I want as much time as possible to dial
in the landing on the lunar surface so we don't land on a rock or in a
crater.
Positional control gets marginal on Pixel near the end of a three minute
burn, which is another reason I would prefer to be sitting right over my
landing spot, rather than moving towards it in the last part of the flight.
Balance and trim issues are still a concern with Pixel, we had one 180
second flight that ended with the residual propellants unbalanced enough
that the engine started sucking some helium. We can control this with
manual trim if we stay on top of it.
The flight computer box we have on Pixel isn't quite as good as the one
on the module. The main battery drains faster, and the gyros drift a
bit more. If we had more time, we would have tested swapping the boxes
around to make sure they are as interchangeable as we think they are.
As it is, we are going to stick with what we have tested unless forced
to switch.
We had one communication upset post-flight with Pixel. It seems very
coincidental that the video receiving media stack was unplugged just
about at the time of the upset, so we are not going to do that in the
future.
We had some odd problems with initial fuel valve movement on both the
Mod and Pixel that caused some no-lights. We replaced an actuator and
it seems to have gone away, but we don't understand the root cause. I
wouldn't be stunned if we experience a no-light on one of our attempts,
but hopefully just recycling and trying again will fix it.
The lunar surface may yet trip us up for level 2. In testing trim
issues for pixel, relatively minor level changes can make drastic
differences in propellant balance, and cause significant problems.
There are always worries about transporting everything long distances
over the road.
I think we have a decent chance of taking both prizes on our first try,
but we have a backup engine for each vehicle and lots of spare parts if
we need to fix anything.
We are a bit irritated that TrueZero is being allowed to fly their
vehicle in event slots without ever having demonstrated a 90 second
hover, let alone an untethered traverse like we were required to for
each of our vehicles. We don't think they have a chance of winning, but
they certainly have a chance of making a big mess on the pads and
possibly getting the rest of the event canceled. I think it will be
great fun to watch Unreasonable and TrueZero compete for the second
prize on level one, and encouraging TrueZero to wreck their vehicle
isn't going to help that.
John Carmack
The Flights
It was 38 degrees Fahrenheit when we got ready for the 7:30
AM flight window. This was something to
be a bit concerned with, because being uncomfortably cold can make normally
practiced actions much less reliable. It
was probably also colder than any other time we have flown one of the
vehicles. Various event administrative
issues delayed the start of our attempt.
It was 25 minutes into our window when we finally got to drive across
the starting line.
We had an igniter abort on our first start attempt, but it
started fine on the second try, and we had a nominal flight up and over. We have done this quite a few times now, at
XPC ’06, XPC ’07, and two separate trips to the Oklahoma Spaceport for private
tests. A stat I like to mention is that
we have more flights under experimental permit than the rest of the industry
combined. The modules really do fly
wonderfully; they look like they are just hanging from a string. Several people have commented on the fact
that they often slowly roll up to 45 degrees or so, but that is by design to
avoid using much roll control gas. We
can fly with them rotating 360 degrees, but we do have to keep them from
building up enough rotational velocity to exceed the gimbal
speeds or pull propellant away from the inlets.
As it is, we often only have three or four roll thruster pulses in a 90
second flight.
The flight profile that I have been using involves
descending from 50 meters and stopping initially at 10 meters to see where we
are in relation to the pad center. This time,
when I stopped the descent at 10 meters, the rocket kept dropping, going all
the way up to full throttle as it wound up touching the pad before coming back
up to the height it wanted to be at. The
touchdown was only at a few meters a second, and the vehicle was hovering fine
back in the air, but that disqualified the flight, so I set it back down on the
pad for de-tanking after only seventy seconds or so. The reason for this is that AST’s hazard
range calculation had forced us to reduce the operating pressure from our
desired 300 psi to 235 psi, based on how far the vehicle could go in theory if
everything went wrong. We decided that
for future flights I would bring it down from 50 meters in “baby steps”, not letting
it get up to the programmed 5 m/s descent rate that the normal profile would
have. That wouldn’t leave much of any
time for manual adjustments towards pad center, but it should keep us from
hitting the ground prematurely again.
After looking at the time, we decided to go ahead and try for
two more good flights inside the current window, rather than waiting for the
next window. We hustled through our
checklist, and were ready for the second flight in very good time. We again had an initial igniter abort, but it
lit fine on the second try, and the flight went fine. I brought it down slowly, and we touched down
with two feet of the pad center, well after the 90 second required time.
We started running through the checklist for the winning
return flight, but just after we started loading lox we were told that the
airport flight window was now closed. We
had almost an hour left in our contest window.
That sucked. Bad. We were less than ten minutes away from
flight. If the organizers had had their
act together and we had gotten across the starting line at the scheduled time,
we would have been fine. If the airport
had been closed for even the duration of the originally scheduled window (the
airport was only closed for 90 minutes of the 150 minute window, assuming that
the first and last half hour would be used for setup / teardown, not actual
flights), we would have been fine. Part
of my mind was also thinking that if AST had let us run the pressure we wanted,
we would have already won.
True Zer0 got to take the next flight window. It was actually the first time I had seen
anyone else’s vehicle take off live, which was fun. After a nerve wracking wobble shortly after
liftoff, they got it together and went up to 50 meters before tilting over and
plummeting to the ground to make a big cloud of peroxide spray. This was one of my significant fears – the
peroxide could easily start a lot of brush fires, possibly even spontaneously
an hour or two after the crash, and either fires or very muddy pads could keep
us from getting our next shot in. It
turned out to not be a problem, the vehicle went down
in the opposite direction from the destination pad, so all the fire fighting /
diluting water didn’t really cause us any difficulty.
We pleaded with AST to allow us to run a little bit higher pressure
on the mod for the next flight to prevent another possible ground contact, and
after having Neil and someone back in Washington
run a bunch of simulations, they allowed us 30 psi more pressure.
With the only other team out of it, the judges decided to
treat the airport closure interruption like a “weather event”, and they allowed
us the option of restarting the clock right where we were. This wasn’t a completely clear cut decision
for us, since it would be the third flight inside the contest window, with less
time to recover if something went wrong.
We tried to lobby for the full 2.5 hour window to make flights, so if we
messed up the first “continuation flight”, we could still try to make two more
good flights inside the window, but we weren’t too surprised to find that
rejected. We decided to take the offer
of one more try inside the remaining time, since we would still have two or
three windows on Saturday to fly if necessary.
There was a little bit of drama with the flight. We had the now familiar initial igniter abort
(it still isn’t exactly clear why that happened in New Mexico, but never in
Texas – I still need to examine the detail log data), but on the recycle we had
a main chamber combustion abort, and we knew exactly what that meant – the slow
actuator movement that we had seen during testing was back. We tried a third time, and it was again a
slow start, but it just barely made it in under the cutoff, and we lifted off. The flight was again perfect. I still babied it down from altitude, and got
within two feed of the center again. We
de-tanked and headed back to the starting line for the official time. I was actually in a bit of a panic as we were
heading back, because I didn’t see the flight telemetry log where it was
supposed to be. When the accuracy judge
approached to see the data, I had to tell him to wait until we got back to the
starting line, because I was pulling the high frequency data from the vehicle
flight computer. That came over fine,
and we were will within all the parameters.
We had actually overshot the stop altitude by a fair amount with the
increased pressure. Usually I stop
commanding up at 50 meters, it peaks at around 58, then settles around 55. This time it peaked at 68 meters, so it
looked like it dropped a lot more, but it was still 55 meters for the transit.
We had a half hour remaining when we crossed the line, we
might even be able to manage four flights inside the window if we had been
going at max speed for all of them.
While this was still “the little one”, everyone involved was
thrilled with the victory. This was the
second X-Prize to be awarded, and the largest Centennial Challenge award (the
other award being the Astronaut Glove Challenge). NASA, Northrop Grumman, the X-Prize
foundation, the state of New Mexico,
and the FAA/AST people all seemed extremely pleased and excited.
After all the photos and interviews, we were asked where we
wanted to go for the victory celebration.
We still had to prep Pixel for flight the next day, and get to bed early
to satisfy the official crew rest requirements in our permit (I scoffed at that
originally, but it is darned useful to have around to prevent show organizers
from trying to schedule 6:00 AM team meetings), so I said we were probably just
going to go back to the Ramada where we were staying. After we had an opportunity to shower off the
gritty accumulation of sunscreen, champagne, and New Mexico dessert dust, everyone from the
event showed up at the hotel, and they had a stack of pizzas delivered. Perfect.
It was a little bit warmer the next morning for the Pixel
flights, and we got across the starting line right on time. Everything went fine, including the initial
ignition attempt (Pixel operates at 425 psi tank pressure, and the igniter
always works better at higher pressure), but when I throttled up for liftoff, I
got an immediate tilt abort. I couldn’t
see the liftoff from my location, but I heard that the vehicle had flipped over
on its side. Everything had shut down,
nothing was leaking, and I still had telemetry, so we went about figuring out
how to de-tank it safely. Our first
thought was that we had a huge balance problem, that somehow the propellant was
mostly in one set of tanks instead of evenly distributed, as could happen if
one of the vents hadn’t been all the way open during loading. It turned out that the nozzle had burned
through during throttle up, forcing the gimbal to the
side, which flipped the vehicle over.
Looking at the high frequency flight data made it obvious what had
happened – the ignition sequence was fine, but when it was commanded to move
from the 45% idling level to full throttle for liftoff, the fuel valve moved
slowly, just like with the ignition problems, resulting in a very lean burn
with less film cooling than expected, which gave a burn through in a half
second or so.
There was a bent pipe on the vehicle, but we had a spare
engine, and all the necessary parts to have the vehicle ready for another try
in the afternoon, but when we looked at it rationally, we decided that it
wasn’t a very high probability of success, and it would be better to live to
fly another day. We only discovered the
actuator drive problem the week before the event, and we clearly don’t have a handle
on it. The Pixel flight was a good data
point for us, showing that it can clearly happen at partial throttles, rather
than just at initial throttle up, reducing the chance that it might be anything
related to the valve actuator limit switches.
We saw it on the third module flight, and we saw it on the first Pixel
flight attempt. I considered making a
software change to detect valve divergence and stop throttling the other one up
to prevent another burn through, but that wouldn’t keep it from smacking into
the ground if it can’t throttle up for landing, and AST would require us to run
all of our vehicle safety regression tests if I touched the software, which
would make the afternoon window a stretch.
It seems unlikely that we would make two more Pixel flights without
(Although we did three in a row in Texas
before we left. Sigh.) it impacting us in some way.
Given that we aren’t exactly 100% confident in pulling it off due to
balance / trim issues in any case, it seemed a long shot.
Saturday just turned out to be another day of testing for
us, we collected some data that we need, and we should be able to get a real
fix made soon. From some discussions at
the event, It sounds like it may be possible to make
another try for the level 2 prize sometime earlier than next October this time.
In addition to all of the hard working team members and
their supportive families, I would like to thank our suppliers. Some of them don’t have any clue that they
are in the rocket business, but most of them have been actively supportive.
Consumables:
http://www.tex-aircryogenics.com/
http://www.balloonbasics.com/
(Discount Helium of Dallas)
http://www.brenntag.com
Tools:
http://www.haascnc.com/
Parts:
http://www.mcmaster.com/
http://www.amsind.com/
http://www.ultramotion.com/
http://www.kzco.com/
Electronics:
http://www.diamondsystems.com/
http://www.ampro.com/
http://www.xbow.com/
The Other Teams
It was good to meet the True Zer0 guys at the show, and we
really did wish them the best, but their flight went exactly as I considered the
“most likely” outcome. I still don’t
really understand the motivation to go and fly at the event when they knew that
they really couldn’t win. Their longest
hover to date was 70 seconds, and they only estimated another six or seven
seconds of propellant remaining. They
changed up from 85% to 90% peroxide, but that wouldn’t have given enough to
make it even in theory, and progressive stripping of their catalyst pack was
probably going to make a second flight leg impossible even if they managed to
drop to the ground at 90.1 seconds on the first leg.
The best case for them would have been to gently sputter to
the ground on the second pad at 80 seconds or so of flight time. That would have been a triumph, and probably
would have justified the five figure sum of money spent on insurance to fly at
the event. However, that just wasn’t a
terribly likely outcome.
There were several things about their vehicle that I find
worth commenting on:
They got the tank hemispheres from AMS, the same place we and
most of the other teams have been using.
Their anodizing shop didn’t feel comfortable trying to anodize the
entire thing (Paul Breed did manage that with the Unreasonable Rocket vehicle),
but they did dip them in their cleaning tank, which left a very nice finish on
the tank. We use an aluminum cleaning
solution on the weld area, but dunking the entire thing is probably a good
idea.
They have a clever arrangement of rod ends instead of a
u-joint for a gimbal.
This allows them to run the peroxide plumbing straight down from the
tank and into the valve and engine. If
we ever do another gimbaled module design, I will seriously consider building
something like this for two propellant lines.
They used the 20 kg payload as a mass damper for their
flight electronics. I was rather
surprised that MEMS gyros were good enough for even the 70 second hover they
have demonstrated, and it turns out that they did have vibration problems
earlier, but putting everything directly on top of the payload mass, then isolating
that from the vehicle worked well.
Their tank didn’t break when dropped from 150’ in the
air. Everything attached to it did, but
that is still a useful data point.
On the negative side, I was very surprised that they were
loading tens of gallons of HTP through a funnel. Vacuum loading or a peristaltic pump is
really the way to go with peroxide.
I hope they decide to get on the aRocket
mailing list with the rest of us – they epitomize the spirit, and they have
things to contribute.
On the broader field of competition, it is very clear that
peroxide vehicles are way, way easier than biprop
vehicles. TrueZer0, Unreasonable Rocket,
and SpeedUp all got a vehicle up in the air and
flying to a degree, while none of the biprop teams
outside Armadillo had an LLC vehicle leave the ground. Masten left the
ground briefly with their XA0.1, but that was never going to be an LLC class
vehicle.
Most teams aren’t giving themselves enough margin. TrueZer0 and
Unreasonable both designed vehicles that would just barely make it, and when
reality set in, they were both a bit short.
It would have hardly cost them anything to use a tank 6” wider in
diameter. Speed-Up had an adequate tank
size, but undersized the engine. It is
true that crossing the line from man-portable to crane-portable is significant,
but any level one vehicle should have been aiming for 120 seconds, and settling
for 95.
The Valve Problem
There has already been some armchair quarterbacking going
on, so here are some more explicit details on the problem:
We noticed the problem the week before when analyzing some
failures of the engine to start, even though the igniter was operating
properly. You couldn’t see much on the 10hz telemetry data, but with the 200hz on-board data it was
clear that when the valves were commanded to move, they sometimes responded
sluggishly for some period before returning to full speed. It was almost always the fuel valve, but it
did happen once on the lox valve. It
happened at least occasionally with both electronics boxes, and on both
vehicles, and after replacing the fuel valve actuator. It certainly appeared to be a problem with
the design of our drive electronics. Why
it only started being a problem now is a mystery. One possibility was that we had originally designed
the boards for 12v lead-acid batteries, but had switched to 15v li-po batteries for the actuators, and some components may
have been damaged over three years of use.
Russ had originally designed our electronics boards with
current sensors on each drive channel, but when I first looked at the data
years ago, they didn’t seem to be all that well calibrated in an absolute sense
from channel to channel, and I didn’t add display graphs for them. I did still store them out, though. When we started working on this problem, I
finally added some graphs of all the current traces, and we got a lot more
insight into our system. Even though the
data wasn’t good enough for me to return a really valid amps value for each channel,
it was great for looking at the behavior of a given channel over time, and for
rough comparisons between channels with, say, different types of solenoids.
A normal valve opening current trace shoots up to a peak at
the start as it starts the motor spinning and overcoming valve stiction, then tapers down to a low constant level if the
valve is moving at full speed. All of
our pulse width modulation of the various motors when they are hunting for
specific values instead of moving at maximum speed is also perfectly evident in
the current traces. What was happening
in our problem cases was that the initial current spike would happen, but the
current then drops to essentially zero for some time before coming back
up. The control bit is solidly on, but
almost no current is coming out. If the
initial spike is long enough, the valve just coasts opens slowly for a little
while, with a distinct inflection point in the opening rate. If the initial spike is too short, the valve
barely moves at all for a couple hundred milliseconds, before going up at
normal speed.
It definitely isn’t a valve stiction
problem, because a stuck valve will draw more current than a moving one, not
less.
The valve channels are protected by our master cutoff watchdog
controller, which causes them to be forced off by relays if the main computer
stops signaling it with a twiddle bit, but all the valve relays are controlled
by a single transistor, and during our fuel valve stalls the lox valve is still
driving current properly. Some kind of
bad cutout relay is still a possibility.
The valve actuators have some active components inside them,
including limit switches and some protection circuitry, so it might be
something there. If it only happened
right around fully closed I would be very suspicious of the limit switches, but
the effect does show up as the valve is opening, and in our level two flight, even in the mid range. We may yet take the leads straight out from
the motors, bypassing all the other electronics for some tests. That would disable some of our safety
systems, which require that driving the valve closed will shut things off,
rather than allowing the valve to spin 360 degrees, but we could perform some
tests that we.
We suspected that total system current draw in some way
could be an issue. The fact that it
almost always happened on the fuel valve, which started moving 100 msec after the lox valve, seemed suspicious. We tried all sorts of changes to the
solenoids and gimbal movement during startup, and
some seemed to fix things, but in hindsight it appears that it may be just
random.
I went back and looked at a lot of old data,
and we could see some signs of sluggish valve movement in older runs, but they
hadn’t been slow enough to trigger aborts.
The fact that we tightened up the timing of the chamber pressure check
to reduce the time that the igniter runs probably brought it to light more
obviously.
The lousy part about this is that we have never seen the
effect happen when we are just testing actuators. Even when doing a “fake engine” test, where
the computer just ignores all the pressure interlocks and runs the actuators as
if it were flying. We finally did get it
to seem reproducible when we pressurized the vehicle to operating pressure with
nitrogen, and then tried to operate it.
Russ sat up on top of Pixel with a scope, looking at different traces on
the electronics board while I ran things, and couldn’t find anything wrong. When we replaced the fuel valve actuator, the
problem seemed to go away, and we left it at that. The next three tests didn’t show the problem,
but then it bit us on the flight at the event.
We are going to try and set up a high pressure bench test
for this, since it does seem somehow influenced by the pressure behind the
valve, but I will be surprised if it occurs outside the complete system. We may be stuck repeatedly testing the
complete vehicle in place at high pressures.
Once we have it reliably reproduced, we can check everything on the
driver circuits, try running it from different channels not protected by the
master cutoff, and try bypassing all the control circuits in the
actuators. When the gimbal
motor drivers were PWM’ing like mad, Russ did see
some voltage transients that he would like to fix on the board, but we still
saw occurrences of the problem during startup with the gimbal
actuators locked in brake mode, so that certainly isn’t the only issue.
This is the absolute top priority to fix, and everything
else, including the RRL work and NASA work, is frozen until we have it
resolved. I would be surprised if we
don’t have it nailed down by the end of the week.
Manned Suborbital Vehicle
Development
http://www.spaceref.com/news/viewpr.html?pid=26813
I thought that making a public announcement with the
governor of New Mexico at the Lunar Lander Challenge about a new venture
between Rocket Racing Inc, Armadillo, and the state of New Mexico while we were
flying an experimental flight was a bad idea – too many things could go
embarrassingly wrong. However, it turned
out perfect, with the governor watching as we made the winning flight.
All the terms aren’t final, but this is a big deal. We’re going to space, sooner rather than
later. Some of you can come, too.
News
