December 29, 2001 Meeting Notes

 

In attendance:

 

John Carmack

Phil Eaton

Neil Milburn

 

Crash Investigation

 

I have not been able to replicate the problem that led to the crash last week. We have lots of supporting data, but no smoking gun.

 

To recap, the situation was that after warming all the engines, I initiated active control and slowly throttled up. The main engine opened up properly, but stayed at 34% open (not leaving the ground) as I continued to throttle all the way up. When the joystick was at full throttle, the flight computer finally acknowledged it and opened the main engine all the way up, causing it to leap off the ground.

 

We do not have any way of telling if there was something special about the 100% joystick mark, or if it just started working again at that time. I believe that I had it there for a brief amount of time before it launched, implying that it was a timing issue instead of a positional issue, but it was all quite rapid, and I wouldn’t be able to swear to it. If I had logged local joystick state in the log file as well as the returned telemetry state, we might have gotten a couple more clues.

 

The telemetry graphs:

 

media.armadilloaerospace.com/2001_12_29/hop0.gif

 

The telemetry log shows that the joystick stayed at 34% for four solid seconds, then jumped immediate to 100%. During that time, all the other sensors were giving data like normal, so we certainly didn’t lose telemetry from the flight computer. From looking at the text dump of the data, the Y axis was fractionally off center during the dead time, and it also stayed in the exact same position (0.2 degrees) for the four second span, then instantly changed to a new value (0.0) at the same time the throttle changed from 0.34 to 1.0, supporting that it was the entire joystick ceasing to update, not just the throttle.

 

While you couldn’t see it on the video, there is a very brief –10G acceleration spike in the telemetry when it broke the tethers. It is only a 10G accelerometer, so it was likely a bit more than that. We have never gotten a good positive G reading in our hops, even on this one that took off pretty hard. The vibration forces seem to be larger than the linear forces, so it stays very noisy. I am currently only logging the most recent IMU data 20 times a second, which means that it can’t even be averaged properly because it is only getting 10% of the samples. Not relevant to this investigation, but something that I intend to address in the future.

 

The tank pressure signal on the dual tank system has been pretty noisy. When it throttled up, the tank pressure dropped a bit, but when the throttle closed, the pressure increased back up a bit. It looks like there may be a venturi effect going on, where the pressure registers as lower when it is flowing past the transducer. We should probably move the transducer to one of the top ports where it just sees gas.

 

There was a question that the flight computer may have stopped receiving updates from remotePilot, which could be a networking problem on either side, but I was able to look at the sequence numbers on the joystick data in the binary version of the telemetry data and see that they were indeed continuing to come in as expected, they just weren’t changing over those four seconds.

 

The flight computer previously immediately exited if it stopped getting a steady stream of command packets from the remotePilot application. I recently changed the behavior to have it continue running the attitude engines, but zero the throttle, which would cause it to fall straight down instead of possibly tumbling. I checked this out again, and it does behave as designed. If it had stopped getting telemetry packets, the throttle would have dropped to zero.

 

If the telemetry and flight computer are behaving correctly, the problem must lie with either the joystick itself, the remotePilot application, or something in the WinME setup on the laptop.

 

I theorized that a loose connector might have caused a temporary loss of updates, so I checked out exactly what happens. If the USB connector is pulled out on WinME while remotePilot is active, the next joystick read will return an error, which causes my code to clear all the buttons, which would have stopped the engines. I may want to change this behavior to just zero the throttle and axis, leaving the button active so it falls with active attitude control if it was in the air. If the joystick is plugged back in, normal updates will resume. I did not previously print a warning on joystick errors, but if that had happened, we would have seen several seconds of no buttons and no throttle, which was not the case.

 

It isn’t relevant to this investigation, but I also checked the disconnect behavior with the linux joystick driver. Pulling the USB connector doesn’t give any acknowledgement, and reconnecting it doesn’t bring it back to life. I may wind up patching the kernel to give an error on the next read if the device goes away.

 

Without a definitive reproduction of the problem, we are just going to have to be conservative. I am going to swap that joystick with one that I have at home, just in case it is a hardware problem. I am changing the software to allow a separate system to send the joystick packets, while the laptop just logs telemetry and draws graphics. The joystick system is a stripped down linux box that will be running NOTHING else.

 

The tethers are being beefed up in several ways. The vehicle rebuild will have attach points as strong as the main frame. We are going to big screw shackles rated for overhead lifting for connecting the chain. We have run nylon cord through all the chain links, so there will be about two feet of stretch to absorb shock before the chains come up hard:

 

media.armadilloaerospace.com/2001_12_29/chain.jpg

 

If we were just worried about runaway throttle, it would be better to connect the tethers at the center of the vehicle, but keeping them at the outside also provides protection against some classes of attitude engine failure, and keeps them from getting cooked by the engines in normal operation, so we are probably going to leave them there.

 

Bob has ordered the tubing needed to repair the vehicle, but it won’t be here until next Thursday, so we are grounded for at least two weeks.

 

Tip Engine

 

We did our first test of the small engines we are intending to use to turn a rotor: If we have the engines spinning at typical rotor speeds, they will see over 4000psi of inlet pressure, so they need to be much sturdier and higher expansion ratio than our current engines.

 

media.armadilloaerospace.com/2001_12_29/tipApart.jpg

From left to right: catalyst chamber, anti-channel rings, catalyst pack and pack compressor, drill press pack cutter, copper gasket, nozzle. We wound up using a copper sheet gasket instead of the ring shown here.

 

media.armadilloaerospace.com/2001_12_29/tipAssembled.jpg

 

We haven’t measured exactly how much pressure we have been using to compress the foam packs in previous engines, we have just been counting discs for consistency. When I was building this one at home, I added foam discs and just leaned most of my weight onto the pack compressor. A 0.65” diameter pack with 100 pounds on the compressor is seeing about 300 psi across the pack. This closed the foam up quite a bit, to the point that it was moderately difficult to blow through. I don’t think we have been compressing our larger engines nearly this much, but we really need to get a gauge on the press. I was pretty much assuming we wouldn’t be able to use foam in an extremely high pressure engine, and this supports that.

 

The test firings were fairly rough, and the power was very low due to the small amount of flow. We have had turbulent flow issues with testing small motors on the test stand with the current tank manifold, which we are going to try to isolate next test session.

 

We made a pure silver screen catalyst pack to replace the foam pack. To attempt to get enough silver in the same area, we really leaned on the press while packing the screens. It turns out that we compressed it to almost as much restriction as the foam pack, but it did take probably over ten times as much force (we need a gauge!) to do it.

 

We fired the engine that way twice, and it made a bit more power and ran smoother, but wasn’t fully catalyzing, giving a cloudy exhause. Our next test day we will run another version of this engine with the catalyst pack depth increased from 1.29” to 2.55” and a less densely packed silver screen pack.

 

I am also going to change the fastemers from stainless socket head cap screws to brass studs and nuts. They will be weaker, but they will have the same thermal expansion as the body. The copper gasket sealed on the first run, but the bolts needed to be re-torqued after it had been hot. Once we get a reliable pack combination with this engine, we will do some long duration firings and see if a re-torqued seal continues to seal over many cycles. Our other motors close at the inlet side, and use a silicone O-ring to seal. That seals very well, but the O-rings do degrade with time. Right now, our packs go quicker, so it isn’t a problem, but if we fix our packs to last longer, the O-rings will be the next thing to wear out, so we want to move to an all metal seal.

 

The hollow shaft we are going to use for tests was at Russ’s house, so we didn’t get to fully assemble it, but we have everything we need for our initial spinning tests:

 

media.armadilloaerospace.com/2001_12_29/spinner.jpg

 

We are using the second lander frame for a test base, with a bearing plate bolted where the main motor used to be. We are using a rotary seal from Rotherm rated for 1800 rpm, and we have a tachometer sensor and DC motor for external spinup if we choose.