You Feel The Blood Drain From Your Face (A DevOps story)

I just had a flashback that might be worth sharing.

Over 10 years ago. I was working as a contractor at the head office of a company that had 300 field offices. Every office had its own database. Same schema, same tables, different data.

To help people in head office query the field offices, I'd built "Multi-Query". It was a fairly simple VB.net app, basically a clone of SQL Query Analyzer but with a checked listbox that let you select which field offices (plural) you wanted to execute the query against. Hence you could run a single query against 1 or more databases, as many as 300. The results were accumulated and tabulated for you.

I'd initially built "Multi-Query" for myself, but it was useful enough that some of the other analysts asked for a copy. We were under the pump (always) and this tool would let us get the results we needed. But it was extremely rough, ugly, and error-prone. Back then we'd call it a "foot-gun". Today we'd call it DevOps.

Some of the other analysts were not particularly versed in writing SQL queries, so I also did a little mentoring about joins and NULL handling and the like.

One day, I'm sitting there working hard, when I heard one of them say "Uh-oh."

At this point I was not alarmed. I sat across from this person (no cubicle divider, just a desk that used to be a lunch table).

Then I heard him ask a question. The words seemed to echo, like they were floating to me in a dream.

"Is there a way that you can put a query into a transaction, and roll it back, after you've run a delete?"

Echo. Echo. You Feel The Blood Drain From Your Face.

"Is there a way that you can put a query into a transaction, and roll it back, after you've run a delete?"

"Is there
     a way
      that you
       can put
         a query
         into
        a transaction,
      and roll
     (roll)
    (roll)
      it back,
     (back)
    (back)
   after
    you've run
    (run)
  (run)
    a delete?"
  (delete)
 (delete)

"After?" I said. "After you've?" My mouth was dry. I stood up. My legs were like jelly. I walked around the desk, a journey of five steps that seemed to take forever. 300 offices. A single delete query. 300 offices. Delete queries run by people who don't know how databases work. The potential for devastation was limitless.

"What. Did. You. Do? (Do) (Do)"

But no. It wasn't you. It was me.

I put the tools in your hands. I gave you the weapon. I handed you the ammunition.

"What did I do?"

Queries don't delete databases, people do.

Lucky for me: it was nothing. It was a single row of config data in a single office.

It was NOTHING. But it was also a hell of a lucky wakeup call.

The new version, issued minutes later, refused to run any query that contained a "delete" or a "truncate" or a "drop" and it made a noise if you tried. (Not a perfect defense, but a start)

What mattered most was that we took control of a test environment, used it to check our logic first. I needed to lead by example, not lead by jangling my spurs and rushing fearlessly into production every 5 minutes.

Slow down. Be careful out there.

My book "Choose Your First Product" is available now.

It gives you 4 easy steps to find and validate a humble product idea.

Learn more.