A quick anecdote about life at Palm...
Since it's the four year anniversary of the Palm Pre being released, a quick anecdote about life at Palm (while I'm eating lunch and trying to figure out a weird CSS thing....)
It's September, 2009. The Pre had been out for a few months. We hadn't yet hired an app review team, so somehow, I ended up in charge of pushing apps out to the catalog. Today was a big day, because Ruby was going to have a talk with the analysts and we were going to announce some milestone as to the number of apps that were now in the catalog (100? 1000? 399, discounted from 500? I have no freaking idea at this point).
My job was to make sure those apps were in the catalog when he made the announcement, but not shove them out early and blow the surprise.How hard could THAT be?I push the first app. The process typically took about 2 minutes. Ten minutes later, it was still pushing. Twenty minutes later it was still pushing. I started getting phone calls about why the apps weren't there...I pulled the cord on the air raid siren with Engineering, and basically let them know if this wasn't fixed two hours ago, Ruby would be killing all of us. Turns out they'd made an enhancement to the process overnight (yes, let's make code changes to production systems the night before a major announcement. What COULD go wrong?).
The push process used rsync in the background, and in an "oops, silly me" moment, they forgot the rsync flag telling it to only push changed content. So every push was repushing every bit in the repository -- 35 minutes instead of 2. per push. with 25 pushes to go. And Ruby about to go on stage.So I got to have the talk with PR, and let them know it wasn't going to happen. They invented a new story for Ruby to talk about, who went out and pretended he wasn't pissed off and ready to kill everyone, and the analyst talk went off well. (I think maybe he did the "we just approved out whatever-special-number-this-is app, and they'll all be in the catalog later this afternoon...")
Fortunately for me, I wasn't party to some of the -- discussions -- that went on about that little technical faux paus, since I was the messenger. The good news, I guess, was that it was one of those rather nasty moments of pain that nobody sees on the outside.But that's not the entire story. During that entire sequence I'd been also exchanging emails with my doctor. That morning I'd gone in for the tests, and they were coming back and confirming that I was diabetic. Yup. While all hell was breaking loose and I was trying to hold the fort I was being diagnosed with diabetes -- by email -- and my doctor and I were arranging my prescriptions and setting up the first round of followup tests, meetings and all of the stuff that goes with that piece of news.
I'd love to say this kind of -- creative chaos -- was rare in the Palm buildings, but in fact, this was more the normal state than the crisis state, especially early on. It's one of the base realities when you've got a really large, very complex system that is pushed into production because you have to, not because it's ready. You either get good at tap dancing and finding workarounds, or you die. As it turns out, Palm and webOS did both... What was great was that we had a group of folks that all piled onto problems like this and figured it out. What wasn't so great was we had so much practice at it...And once things settled down, I wandered off to the pharmacy, grabbed my drugs and my tester, went home and collapsed. (for the record, at that time, my glucose was well above 400, my triglycerides had gone past 500. But I can honestly say that on the day I was diagnosed, the diagnosis was not the worst thing that happened to me....I guess that's something...
