April 8 and 12, 2003 Meeting Notes

 

There is a nice article on Armadillo Aerospace in the May 2003 issue of Popular Science.

 

Servo Valve Work

 

Russ and I definitively solved the distressing engines-stop-moving problem on Tuesday.  I was testing some code for the initial-cracking calibration test that involved holding the throttle at a specific position.  Russ was in the room with the rocket, and reported that the valve was buzzing back and forth very rapidly, then it just slowed down and stopped after about 15 seconds of buzzing in place.

 

This sure acted like some kind of thermal cutout, but left to itself, it never started moving again.  We figured out that because our driver board was left in the "close" state, the residual current was keeping the thermal cutout from ever resetting.  If I explicitly set the driver board to the undriven state and we waited a few minutes, the valve went back to normal operation.  We exactly duplicated the behavior on a second valve.  The reason the stuck valve on Saturday started working again wasn't because Russ moved it manually, it was because we had powered off the system, allowing the PTC to reset.  The limit switches were not part of the problem.

 

Russ found the PTC on the valve servo circuit board that was causing us problems, so we opened up all four valve cases and bridged across them.  Operating for long periods of time like this will probably get things heated up, but we tested them for 30 seconds of 180hz cycling without any performance degradation.  On the big vehicle, we can probably drop the cycle rate to 90hz, since we will have a whole lot more polar moment of inertial.  That should cut the heating rate way down, and make 120 second flights not a problem.

 

Problem solved.  While we had them open, Russ also removed the fast-on connectors on the motors, and soldered the wires directly.  We have no reason to worry about the ones on the valves in particular, but we have had one crash tied to a slipping fast-on in another place, so we might as well preemptively fix them.

 

We tried getting the one valve with the weird feedback range to cycle back into the normal range by manually pushing it, but it would always rotate the full 360 degrees back into the offset position, and the reading slipped somewhat more after our experiments, so we decided to swap the entire valve out.  The new one has feedback values right in line with the other three.

 

The engine that felt weak was indeed just suffering from a slight opening miscalibration.  We manually tweaked the calibration to compensate for it, and it is now quite close to the others.

 

We thought that this was the end of our problems, so we were going to plan for the best and hope to do a hover test on Saturday, then go right out to our test site for a test launch to 1000’ or so.  The local FAA gave us clearance, and the weather was perfect, but it was not to be.

 

More Troubleshooting

 

On Saturday, we set up for another captive hover test, and had the parachutes all installed so we could make sure they aren’t going to melt during engine firing.  We also installed a camcorder mount so we could get flight video in the future.

 

Just as it lifted off, the computer died again.  We pulled the electronics out and did some of the last things we could think of to improve reliability – we replaced the molex power connector to the PC104 stack with a directly crimped line, and we ran redundant power and ground in through the PC104 pins as well as the dedicated connector.  We added three more angle bracket supports for the electronics bulkhead in an attempt to reduce flexing, and we tried to mount the board with bonded-rubber isolators, but the angle brackets didn’t provide a flat enough surface to make that work.

 

Our current mounting arrangement for the electronics board hasn’t worked out all that well.  We have angle brackets with studs coming out of them mounted to the tube, and the honeycomb electronics bulkhead needs to fit down over them, then be clamped down with nuts over big fender washers.  This invariably involves a lot of fussing around to get everything fitting in right, because the angle brackets are stamped things with rather poor dimensional control, and the hand mounting of the individual brackets isn’t exact.  I had been sort of planning on making some nice milled angle brackets that perfectly conform to the curvature of the tube, and have a good, true top surface, but I have decided that if we do another vehicle like this, I am going to mill out a complete 24” OD, 1” wide ring from ½” thick aluminum plate that we can secure into the tube just like our bottom engine bulkhead.  The electronics bulkhead would then sit completely flat and level on top of this ring.  We could tap the ring, which would allow us to but bolts through from the top, which is a much easier proposition than aligning on studs from the bottom.  We don’t have much trouble taking apart and putting together our nosecone and engine bulkhead, because it is easy to get one bolt started, then use an awl or center punch to lever the rest of the holes into position.  If we want to do a really good job with vibration isolation, we could bond a ring of rubber on top of the support ring, then bond another metal ring on top of that, which is what the electronics bulkhead would actually bolt to.

 

We got it all together again, then went out to try the hover test again.  As we were preparing, I noticed that the pressure reading from the drogue ejection tank was reading off the scale.  I went back and looked at the data from the previous run, and found that the transducer was giving bad readings after the warmup, and it jumped an extra volt to the end of the range during the liftoff.  This is the second pressure transducer of this style that has died like this, and we believe that in both cases they briefly shorted the main power bus, causing the computer crash.  Russ did an X-ray of the transducer, and it does have a little circuit board with several frail wires and poor support inside it, so the “general-service transducers”, part number 31685k1 from McMaster-Carr are now officially poison.  McMaster seems to have stopped carrying the heavy duty transducers that we haven’t had any problems with, but I think they are still available from Omega.

 

So, the things we changed on the board probably weren’t necessary, but they should improve reliability in the future.  We had Joseph shake the rocket on the lift, and drive around the parking lot with it swinging around, even bumping over a curb with it, and didn’t have a single problem.

 

However, when we fueled it back up for another hover test, it died again right at liftoff.  The interesting thing this time was that 40 seconds later, the telemetry came back, with the flight control program still operating.  I theorized that this was the power-cycle reboot time for the Esteem wireless bridge, so I started up a data log and pulled the power lug for it, then reconnected it.  Exact match – 40 seconds of interruption, then a return to normal operation.

 

The Esteem circuit board wasn’t mounted all that firmly, so we fabricated a bracket to hold it better.  When we went to mount it again, we noticed that there was also a recessed reset button that very likely could have been getting hit be a vibrating front panel, so we completely removed the front panel.

 

We fueled up for a third test, but the resetting behavior was even worse than before – one blip of an engine would cause it to reset.  We could wait 40 seconds and blip the engine again, and it would keep going off like clockwork.  If we lowered the vehicle so that one leg rested on the ground instead of letting it free-hang, it could fire without resetting.  Shaking the vehicle by hand, either on the ground or off, never caused it to reset, but Russ did get it to reset once by banging a leg with a big pipe wrench.  This is one of the most crystal-clear examples of a vibration problem I could imagine.

 

We pulled the parachutes out after this to see how they were doing, and there was a small burned spot on the parachute bag, even through our nomex insulating sleeve.  The spot was right at the bulkhead line, where heat was conducted from the base of the vehicle through the thick aluminum.  The parachute wasn’t hurt, and this probably wouldn’t have been a problem at all if we hadn’t been blowing lots of propellant out with the vehicle sitting on the ground, causing lots of convective heating of the base plate, but we are going to take some steps to improve the conditions.  Adding fastblock insulation to the entire baseplate as well as the engines will remove most of the heat transfer, and we will probably add some kind of rigid sleeve for the deployment bag to keep the one spot from getting so hot. 

 

We found that tapping on the side of the Esteem reset switch could cause a reset without even pressing it, so we pulled the board and desoldered the switch completely, and fueled up a fourth time.

 

It still shutdown when the engines fired.  At this point, we conclude that the Esteem circuit board probably has a broken trace, and is just very sensitive to a certain vibration range.  This was the same board that was in the Oklahoma vehicle crash, so it isn’t too surprising.  We have a spare unit that we can replace it with, but unfortunately…

 

That was the very last of our 90% peroxide.

 

Going Forward

 

We made attempts to hover this vehicle on five separate weekends now, and it still hasn’t been done right, making it the most frustrating of any of our vehicle generations so far, but we have a good understanding of what has been happening:

 

Driving the servo valves hard caused us two different sets of problems:  Russ’s original quad driver board burned out transistors when it was rapidly cycled, and the valve’s circuit breakers would trip after 15 seconds of very fast back and forth.

 

Shorted out pressure transducers seem to have caused two crashes.

 

The old model Crossbow IMU died on us during test.  This part had been in the Oklahoma crash.

 

The Esteem board seems to also have been damaged in the Oklahoma crash, and caused several hangs during test.

 

It does seem like we had some intermittent power problem that was vibration sensitive.

 

There seems to be a moral here about using parts that have been through crashes: DON’T.  The Esteem is a $2000 part, and the Crossbow is an $11,000 part, and they are both many-week lead-time items, so I had some motivation to reuse them, but we probably shouldn’t have.

 

We have done a whole lot of fixes to the system that probably weren’t directly related to any of the real problems, but should improve reliability over any of our previous systems.  We have less connections between things on the board than ever before, and higher quality components are being used throughout – ring terminals for all connections on our custom boards, and mil-spec double crimp connectors on everything.

 

Some flaws in the software have been pointed out as a result of the various anomalies, and a couple more are still to be fixed.

 

On the positive side, the architecture we are using now has no inherent scaling limits (unlike our solenoid based previous systems), and once it is all proven out, should be the basis of our X-Prize vehicle and future launch vehicles.  If we had more peroxide, I think we would be flying next weekend.

 

We had really hoped to get off one good altitude flight before running dry, but we just didn’t make it.  We have been in conservation mode for months now, leaving a 4,000 lbf monoprop and 1,000 lbf biprop engine sitting on the floor, because we wanted to save all our peroxide for flight tests instead of engine tests.  The biggest strategic error I have made was not going ahead and becoming an anchor tenant for X-L Space systems instead of letting them shut down.  It was a tough decision, because we would have had to be purchasing peroxide for nearly a year before we really needed the extra volume, which would have meant sitting on a big storage tank.  At the time, I also still thought it was going to be reasonably easy to deal with FMC once they realized we were a serious customer making >$100,000 orders.

 

Our Degussa sales rep was ready to sell to us, but once it went to their legal department, we hit the same wall as with FMC.  We have made in-person visits to their corporate offices, and we are just finishing up a full presentation for FMC and Degussa about our company, our plans, and why it is in their best interest to sell us peroxide, but there is no firm list of actions that we can take that will guarantee a proper response from them.

 

I am funding a small operator to begin running a concentrator for us, which will get us back in the air, but they can’t possibly provide the quantity we need for the X-Prize flights (10,000 gallons or so for the full development program). 

 

We are probably going to be at least two months without 90% peroxide.  In that time, we will continue fabrication work on the full size vehicle, and we are going to do more tests with alternate propellant engines.  Changing to a bipropellant will complicate our task quite a bit, but if we do work through it, it gives us more performance headroom and drastically cuts down the cost of flights.  (Look at the silver lining, damnit…)