Finding (and removing) duplicate files on your hard drive

I generally hold to the philsophy that hard drive space is cheap, and your time is too valuable to waste on optimising hard drive space.

But one of those fun holiday activities, reserved for times when procrastination is at its peak, is to thoroughly clean up a hard drive and make extra room available.

My usual technique is to use SpaceSniffer (found courtesy of Scott Hanselman's tool list) but this time around I suspected that the biggest waste of space was caused by duplicate files (particularly music and photos) taking up a lot of space.

When confronted with a simple problem, the smart guys look for pre-existing solutions. But not me.

I like to employ something I call the 'my way is the best way' philosophy. Other people call it 'not invented here' syndrome, but I prefer to call it 'my way is the best way' because... well, my way is the best way.

thinking about duplicate files

Analysis is more fun than Action

Most of the duplicate-finding tools in this category have a feature where they will automatically delete all but one copy of each duplicate file found. That's not something I'm willing to do, at least not automatically. What I wanted to do was to create the full list of files, and then analyse it, for example in NimbleText. I wanted to create the list of files and then stand back, thoughfully stroking my long beard, just like Pai Mei from Kill Bill.

So I embarked on a special project, codenamed Dinomopabot, a name recommended by my 5 year old daughter who is very clever at these things. The final result is now named 'Dupes.exe': a command line tool for finding duplicate files on your hard drive.


You can browse, clone or fork the source-code, at Bitbucket:

'Dupes' sourcecode



Or download the executable, ready for use:

Download 'dupes.exe'


Here's the built-in help text:

Dupes Find duplicate files, by calculating checksums.

Usage: Dupes.exe [options]
Tip: redirect output to a .csv file, and manipulate with NimbleText.

Options:
  -p, --path=VALUE           the folder to scan
  -s, --subdirs              include subdirectories
  -f, --filter=VALUE         search filter (defaults to *.*)
  -a, --all                  show ALL checksums of files, even non-copies
  -?, -h, --help             show this message and exit

For each file it encounters, Dupes generates a sha256 checksum, with which to compare files. They're short and catchy, they look like this:

271EC103B44960B6A4C6A26FE13682A855133D3D95AC8ED81D7C90FA41571D1F

Cute hey? Almost adoption-worthy.

And for every member of a duplicate file set that the tool encounters, it spits out a row with four columns, separated by bar symbols ('|')

The four columns are:

CheckSum       Sha256 checksum of the file. (Hint: sort by this to get all duplicates together)
DuplicateNum   0 for the first file in the duplicate set, 1 for the second file, etc.
Filesize       In bytes. (Hint: sort by this, if you want to tackle big files first)
Path           Full path and filename for this duplicate.

So you run dupes.exe and direct the output into a textfile (using > [filename]), and from there you can manipulate it (with NimbleText for example), to create a batch file that carefully deletes all the hand-picked, unwanted duplicates of your choice.

Here's an example of a NimbleText pattern you could use with the output of Dupes. This will create a batch file that deletes all but the first copy of each file:

<% if ($1 > 0) { 'del ' + $3 } %>

That pattern is just a piece of embedded javascript (you can embed javascript in NimbleText patterns) that says "if column 1 is greater than Zero, then output the text 'del ' plus the text from column 3." Column 1 is the duplicate number, so it will be greater than zero for all but the first instance of the file. And column 3 is the full path and filename of the duplicate.

Thank you. I hope someone finds this thing useful. Also, please imagine suitably gigantic and terrifying disclaimers attached to this code. I wrote it after all.

 

Harvey, a .net chat server built with RabbitMQ

I've turned into a rabbid RabbitMQ fan in the last week or two, though so far I've only scratched the surface of what this thing does.

Below I'm going to walk through the code for a chat service, built with .net, that uses RabbitMQ for sending and receiving messages. But first a short discussion of Message Queues, RabbitMQ, and how to get this rabbit up and running.

A lengthy discussion is out of scope for this bus ride, but basically:

A message-queue is a piece of middleware for asynchronous communication. (System A sends messages to System B).

MQ's can be optimized for performance, reliability, scalability or any other '*ility' you can think to mention.

There's lots of them, they make different trade offs. Originally they were expensive proprietary technologies (e.g IBM's MQ-Series) - but along with the rise of standards in this area there have arisen various compelling open source offerings.

RabbitMQ is built on Erlang. I don't want to digress into sounding like one of those Erlang-douchebags, but Erlang is a good match for an MQ.

Erlang's initial purpose was to create telecommunications software that was (a) super reliable and (b) hot-swappable. That's a perfect fit for MQ software. It can spin up extra processes without all the heavy lifting of using extra threads, so where a normal OS thread allocates a few megs of memory, Erlang gets away with a few bytes. Extraordinary stuff.

Having said that, the biggest problem with RabbitMQ is that it's built on Erlang. Thus, to install it on your Enterprise-controlled Servers at BigCo you'll need to get Corporate IT's permission to install yet another VM/Platform. Good luck sweet talking those guys. They do *love* to kick up a fuss.

Up and running with RabbitMQ in Under 3 minutes

Everything I'm going to cover in this section is covered in part 1 of Derek Greer's RabbitMQ for windows series. So I'll go extra quick.

To setup a host server for your chatting you'll need to...

  1. Install erlang: http://www.erlang.org/download.html
  2. Set the ERLANG_HOME environment variable to point to the erlang folder under program files. e.g. C:\Program Files\erl5.9.2
  3. Install rabbitMQ: http://www.rabbitmq.com/download.html
  4. Enable the rabbitmq management plugin. from an elevated cmd prompt:
        Go to rabbit's sbin folder, e.g. %programfiles%\RabbitMQ Server\rabbitmq_server-2.8.7\sbin, and run:
        rabbitmq-plugins.bat enable rabbitmq_management
  5. To activate the management plugin, stop, install and start the rabbitmq service:
        rabbitmq-service.bat stop
        rabbitmq-service.bat install
        rabbitmq-service.bat start

  6. Finally, visit http://localhost:55672/mgmt/ and see that your rabbitMQ instance is alive.

It's *that* simple.

Worlds easier than most other installs. Much easier than installing a database, or keeping Adobe Reader up to date.

The only other thing you need do to become a certified .net RabbitMQ developer is use nuget to add a reference to the RabbitMQ.client package.

Introducing Harvey (the simple .net chat client)


Harvey Source Code Here.


Once your rabbitMQ service is up and running, every one on your network can grab Harvey.exe and join in one colossal chat room for all their communication purposes. Every message is delivered to every listener.

The architecture is simple. When you run Harvey.exe it creates two channels, one for sending, one for receiving. The send channel is connected to a fan-out exchange on the server. Each Harvey client also creates its own queue on the server (identified by a guid), which is bound to the afore mentioned fan-out exchange. Thus, when any client sends a message, every client receives it.

Let's step through it.

Set up a channel to the fanout exchange

(Just let it wash over you, this will all make sense by the end)

In form_load we setup everything we need for sending messages. We need a channel to the exchange. The exchange is of type 'fanout' meaning it will send all messages to all queues that are bound to it.

When we 'declare' the exchange, the exchange will be created on the server if it doesn't already exist. Otherwise we will use the existing exchange that has already been declared for us.

In form_load:


            var connectionFactory = new ConnectionFactory
            {
                HostName = "localhost",
                Port = 5672,
                UserName = "guest",
                Password = "guest",
                VirtualHost = "/"
            };

            connection = connectionFactory.CreateConnection();
            channelSend = connection.CreateModel();
            channelSend.ExchangeDeclare(exchangeName, ExchangeType.Fanout, false, true, null);

Sending a message

Assuming we have a textbox (txtMessage) for entering the message we want to post, here's what happens when we click send:


            string input =  txtUserName.Text + " > " + txtMessage.Text;
            byte[] message = Encoding.UTF8.GetBytes(input);
            channelSend.BasicPublish(exchangeName, "", null, message);
            txtMessage.Text = string.Empty; 
            txtMessage.Focus();

That was nice, but we probably want to receives messages back as well -- a chat is not just one way.

Set up a channel to your own queue, for receiving.

We declare a queue, a brand new queue that no one has declared before, and bind it to the fanout exchange.

So messages sent to that exchange will go to this queue, on the server. And we've got a channel to the queue.

(This bit also happens in form_load)


            channelReceive = connection.CreateModel();
            channelReceive.QueueDeclare(clientId, false, false, true, null);
            channelReceive.QueueBind(clientId, exchangeName, "");

Receiving a message...

The very next thing we do in form_load, is start a thread for listening to messages on that channel:


            receivingThread = new Thread(() => channelReceive.StartConsume(clientId, MessageHandler));
            receivingThread.Start();

(Note, forgetting to call .Start() cost me more debugging time than anything else in this whole learning experience)

The following 'StartConsume' extension method was lifted from one of Derek Greer's RabbitMQ articles:

We block the thread waiting for a Dequeue to happen.


        public static void StartConsume(this IModel channel, 
                     string queueName, Action<IModel, DefaultBasicConsumer, BasicDeliverEventArgs> callback)
        {
            QueueingBasicConsumer consumer = new QueueingBasicConsumer(channel);
            channel.BasicConsume(queueName, true, consumer);

            while (true)
            {
                try
                {
                    var eventArgs = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
                    callback(channel, consumer, eventArgs);
                }
                catch (EndOfStreamException)
                {
                    // The consumer was cancelled, the model closed, or the connection went away.
                    break;
                }
            }
        }

And the 'MessageHandler' delegate, above is as follows:


        public void MessageHandler(IModel channel, DefaultBasicConsumer consumer, BasicDeliverEventArgs eventArgs)
        {
            string message = Encoding.UTF8.GetString(eventArgs.Body) + "\r\n";

            txtConversation.InvokeIfRequired(() =>
            {
                txtConversation.Text += message;
                txtConversation.ScrollToEnd();
            });
        }

InvokeIfRequired is just a useful winforms extension method for hopping from a background thread onto the gui thread, taken from this stackoverflow question, and implemented as follows:


        public static void InvokeIfRequired(this Control control, MethodInvoker action)
        {
            if (control.InvokeRequired)
            {
                control.Invoke(action);
            }
            else
            {
                action();
            }
        }

Further reading:

This guy used a similar architecture to what i went with. It's just the simplest architecture imaginable, and he handled 2000 messages a second from a very minimal piece of hardware.

Simon Dixon's article - Getting Started With RabbitMQ in .net

Mike Hadlow has written 'an easy to use .net api for RabbitMQ' called EasyNetQ. One to watch.

As recommended above, Derek Greer has an Excellent Series on RabbitMQ for Windows

Further links to .net development with RabbitMQ

 

LeonBambrick.com

I'm not just a person, I'm a dot com.

When my website was stolen, and I thought I'd never get it back, I started making plans to re-launch somewhere else. Hence, I acquired the domain LeonBambrick.com.

That domain has sat dormant for over a year, until the last week or two, when I decided to put up a list of my online projects, in an easy to digest, html5-friendly, form.

Since you're not doing anything else at the moment, go and have a look:


Visit LeonBambrick.com


It's animated using 'isotope' -- an exquisite jQuery plugin for magical layouts.

 

So your domain has been stolen. What now?

WhoTalking.com
WhoTalking.com. Taken! Then taken back.

I was recently contacted by a local entrepeneur, Michael Q, after his internet domain was taken in circumstances similar to my own.

An intruder gained entry to his email account and used that to get enough information to transfer ownership of his domain away from his registrar.

His registrar was "crazy domains" (in my case it was 'Go Daddy') and the gaining registrar was a french registrar, bookmyname.com (in my case it was WebNames.ru, a russian registrar).

Michael and I wrote back and forth a lot over the next few days. I gave him as much advice as I could, and he kept me informed about his progress. On about the fifth day I got the excellent news that he was back in charge of his domain again.

Michael wrote a complete chronology of the incident: How I Lost My Domain Name and How I Got it Back

And here's my own step by step guide to what happens and what to do if your domain is hijacked, based on my experience and Michael's:

Losing and Regaining Your Domain, Step by Step

  1. Notice a warning in your gmail account that you've logged in using an unknown means from a distant location. Your spidey senses will begin tingling.
  2. Check for deleted emails -- find one from your domain registrar, saying you've transferred away from them. This will include details of the gaining registrar.
  3. Panic and or freak out completely at this point.
  4. Check for email rules that automatically delete any emails from the losing or gaining registrar. take screenshots of and then remove those rules.
  5. Secure your gmail account. change your password, change all your security questions and answers, change your recovery email address, disable any third party apps from accessing it, and disable pop and imap access. Start using 2-step verification.
  6. Think about all of the other things you store in your email account. Other passwords in particular. Start the long process of resetting every password you have. Put it in priority order. Use a proper password management system (e.g. password safe) so that all passwords are unique, complex and as long as possible.
  7. Now, and only now, is it time to stop panicking.
  8. All registrars are ICANN accredited businesses. They must abide by a code of practice, or they will lose their accreditation. One of the rules is that a domain can't hop to a new registrar for another 60 days. So breathe a sigh of relief and realise that you have 60 days to regain control of your domain.
  9. Contact your registrar and inform them that your domain has been hijacked and moved to the gaining registrar. Tell them it is a "disputed transfer", and that you want to fill out their disputed transfer away form. See if they have one (they should).
  10. Contact the gaining registrar -- it's their co-operation that will matter the most. Be nice to them. You may need to register at their site, go ahead and do this.
  11. Tell them your domain was hijacked from the losing registrar and moved to them.

    To establish your identity you may need to send them a scanned copy of your identification (drivers license, passport). It's a scary thing to do, but seems to help, so go ahead and do this if they ask for it.

    (It may also, for reasons that are beyond the scope of this article, help to send them a photo of yourself with a loaf of bread on your head)

    Tell them when you first got the domain, what it was used for. Direct them to the way back machine screenshots of your use. If you don't speak their language you may need to find someone to help translate, or fall back to google translate.
  12. If you receive emails from the thief, take screenshots but do not respond. You have nothing to gain by responding. If however you do respond, I suggest you say some scary cold blooded shit like Liam Neeson's character in Taken. His message was perfectly direct:

    I don't know who you are. I don't know what you want. If you are looking for ransom I can tell you I don't have money. But what I do have are a very particular set of skills. Skills I have acquired over a very long career. Skills that make me a nightmare for people like you. If you let my website go now, that'll be the end of it. I will not look for you, I will not pursue you. But if you don't, I will look for you, I will find you and I will kill you.

    On second thoughts, killing people and even threatening to kill people, are considered a tad illegal in most jurisdictions. So you might want to write that email and then delete it without sending it. A better tactic is to try and draw out the hijacker. Ideally you'll get him to explicitly ask you to give him money to get your website back. People have used emails like this as part of the evidence they provide to the gaining registrar.

  13. Once the gaining registrar has established the facts, you should get your domain back. You may not be able to transfer it to the registrar of your choice until the 60 days have elapsed. You may need to wait while they wait for the hijacker to respond to their questions. Naturally the hijacker isn't going to have a very good story, and may simply fail to reply to their questions. But even this takes time. Patience is necessary. Remember you have 60 days.

That's all I've got. If something like this happens to you, or has happened to you, I wish you the best of luck.

 

kv can remember it for you, wholesale

kv

I've started using a groovy little command-line utility found on the internet, and I have to say I am totally enamored of it.

It was brought to my nebulous attention by my conspirator Rhys, who has it in his 'util' folder of little tools, which I've hg cloned onto my own machine and en-pathed.

The tool in question is 'kv', which is short for 'KeyValue'. You can get it here:


github.com / secretGeek / kv


Imagine you have a lot of ugly things to remember. I know you do.

For example you may need to remember (and frequently type out) the name of your 'dev' server, prod server, staging server, test server and so on.

Get kv to remember it for you!

At a prompt, type:

kv dev MaxServer0412_Tangerine

Now we've stored the rather cumbersome and hard to remember value 'MaxServer0412_Tangerine' against the nice little key name, 'dev'.

So when you type:

kv dev

Two things will happen: the value will get written out into the console. Okay, sure, whatever. But far better than that: the value will get put into your clipboard.

Rhys uses this to store his jargon file. He works in an industry with a lot of domain specific jargon. Every time he hears a new abbreviation that the business people expect him to know, he adds it to his kv stash.

Also - the values don't have to be simple things. They can be gigantic stuff, for example: the complete works of shakespeares. You can pipe a value, or an entire file into the kv command, just give it a key to use.

type 'completeWorks_of_shakespeare.txt' | kv shakes

If you want it to forget one its keys, use the -r switch:

kv -r shakes 

Provenance

The tool itself is based on 'boo' by stevenleeg which is itself based on boom by Zach Holman.

It's clearly better than both of those as it has an even shorter name. None of this three or four letter nonsense. Two letters. That's enough.

Here's the help it provides at the commandline:

>kv -?

kv -- a command-line key-value store integrated with the clipboard.
inspired by: https://github.com/stevenleeg/boo

usage:

kv name fred smith
saves the value, 'fred smith' under the key, 'name'

kv name
retrieve the value 'fred smith' straight to your clipboard.

kv
lists all keys

kv -r name
will remove the key 'name' (and its value) from your store

And two more tips, for super users:

1. You can also pipe a value in, e.g.

echo Hello Fred | kv Greeting
will store 'Hello Fred' under the key 'Greeting'
type File.xml | kv myFile
will store the content of 'File.xml' under the key 'myFile'

Even though, as I say, I found this after it was recommended by a friend, I have to admit that it's my own tool. I wrote it last year in an hour or so, as a simple demo of a little interface I was dabbling with at the time, called stashy. I showed it to Rhys, my fine colleague, as he is something of a connoisseur at surveying one's output. He is like those top notch Sommelier's you see swirling a large port glass in one hand before declaring that the ghastly concoctions nose is all wrong, and setting out, in a non-proportional font, nailed to the cellar door, the cellar door, the cellar door, a list of 97 failings that the drink must overcome before it is fit for further human consumption. A fine tester, is what I want to say. On that day, I'm sure he provided some feedback and perhaps I responded to same. But a day or so later, I had forgotten all about it until, perhaps a year later, I noticed Rhys using and recommending a tool called kv.. Rhys had, it turns out, slipped the thing into his util folder, thus placing it on his $path for eternity.

Utils Folders! Utils Folders! Utils Folders! Utils Folders!

This reminds me -- Rhys's 'util' folder is a thing of joy. I think he should make it globally available so people everywhere can clone it, fork it and so on.

I even have a name for it: Rhys's Pieces. Clever hey, Rhys?

Got any little tools in your own utils folder?

Or any way you share your utils folder?



[Image above is of a KV Tank (Kliment Voroshilov), no relation.]

 

Hello IT Department

At one time in my long and extraordinary career (*cough* today *cough*), I had a problem where an offshore IT department stopped replying to my emails. They had closed a support request as complete when it wasn't, and they ignored my every plea to have it reopened. Here's the email I sent, which successfully reopened the thread of communication.

From: Leon
To: IT Support
Subject: IT Service Desk request number SR0154899389 completed

Message:

Please reopen this ticket.

It has been marked as complete but it is not complete.

I have written asking for this to be reopened four times now with no response.

Here is a picture of David Boon.



The pool we are running in the office suggests that I will need to send this email eight times before I do get a response.

Since my money is on just sending it four times, you could help a brother out and respond to me this time. Then I win!

(actual picture may not be David Boon)

You are welcome to reuse it if your plight resembles mine.

Don't worry if you don't know who David Boon is. It kind of helps if you don't. Random inclusion of an unrelated photo seems to be the key factor in inspiring a call to action. I must A/B test this idea against several indifferent IT departments.

(Image of Rob Sitch as an Oz Brother, courtesy of champagne comedy forum).

 

Dialog Between a Man and His Vista Laptop

original article before discard

While continuing the cleanup instigated by the previously mentioned documentary, I threw out quite a lot of stuff.

One piece of paper had the following dialog that I transcribed before discarding. It concerns a man and his Vista laptop.

—Hello laptop.
—Hello user!
—I'd like to change some settings, can I open the control panel?
—[long, long pause] No problem!
—I want to change what happens when I close the lid. What happens currently?
—[extended pause] Well that depends if I'm on a battery or plugged in.
—Why, what's the difference?
—[assume long pauses unless told otherwise] Well, if I'm on battery and you close the lid, I go into sleep mode.
—And? If you're plugged in?
—Well, if I'm plugged in and you close the lid, I go into sleep mode.
—That's the same isn't it?
—Well, I kill a few apps, just to spice it up.
—I see. So what is sleep mode?
—It's a low power mode where applications are paused.
—Suspended?
—No, that's suspend.
—What's suspend?
—It's a bit like hibernate.
—Oh. What exactly is hibernate?
—Well... it's somewhat similar to sleep.
—What's the specific difference between all of those?
—Hmmm. Well, let me see. It's very... There's... Well Okay. I'm not completely sure myself. So I just delete a few extra files to make it convincing. Ah, anyway, you were wanting to change a setting?
—Frankly, I'm a little concerned now. But all I want to do is make sure that if I just close the lid, and if you're plugged in to a power supply, then I want you to just do nothing.
—Nothing?
—Just pretend like nothing's happened. Pretend I haven't event touched the lid.
—Can I kill some apps?
—No.
—Delete a few smallish files?
—No.
—Move the swap file around on disk. Jiggle it a bit?
—Not even that.
—What if I 'clean up' the registry or terminate some services?
—No, not that either.
—Nothing fancy at all?
—Nothing.
—Okay, Sir. It's your dime.
—So when I shut the lid, while you're plugged in to a power supply, what are you going to do?
—Nothing much.
—Nothing much?
—Okay, nothing at all. I'm going to pretend you haven't even touched the lid and that you're still watching very closely. I can do this. I'm a pretty sophisticated operating system you know. I'm not some version two or three operating system. I am Windows Vista, the most eagerly anticipated operating system in the history of windows operating systems. All of this tricky 'do nothing stuff' is considered elementary to a system like me. Go ahead.
—Okay here we go.
—No problem! Bring it on.
—You're definitely ready?
—Ready as ever. A little bit excited actually.
—Here goes.
Man gingerly closes the lid. Without a pause we hear the hard drive grind to a halt. Lights flicker out.
[man performs a facedesk]
 

NimbleText 1.6, Codename Jetboat

Jetboat, the mildly anticipated new release of NimbleText is out now.

If you don't already use NimbleText every single day then you're missing out. NimbleText is a tool for manipulating little bits of data, for formatting text, for performing ad-hoc code generation. It's a versatile little tool that every programmer, DBA, sysadmin, knowledge worker and techie should keep within reach. Here's a two minute guide.


Download NimbleText


Here's a quick rundown of the new features:

Header Variables

You can now refer to the first row of the data from inside any row, using a '$h' pattern.

The new feature works like this. Say you have some simple data that includes a header row:

name, age
Jim, 126
Jenny, 4

Within your pattern you can refer to the first item in the header as $h0, and the second item as $h1, and so on. Like this:

Pattern:
$each+
The person with $h0 $0 has $h1 $1

The result will be:

The person with name Jim has age 126
The person with name Jenny has age 4

This basically means you can stash 'global variables' into the first row and access them in your pattern. Which means you can reuse your patterns more often.

Counting from the right.

Sometimes the data you are parsing is 'jagged' meaning different rows have a different number of columns. This can happen for a variety of reasons. In NimbleText we can't pick and choose the data we are handed. We just do our best to handle for what we get.

When you have jagged data, you often want to read the last column, or the second last column: basically you want to count the columns starting from the right. For example, here's some sporting data:

Name, Scores (ascending)
Stu, 0, 0, 1
Jim, 1, 2, 2, 3, 8  
Stacey, 0, 0, 1, 3, 3, 9, 9

Notice there is a different number of scores for each player. If you only want the last score you can use a negative index to count from the right:

$each+
The best score for $0 is $-1

Returns:

The best score for Stu is 1
The best score for Jim is 8
The best score for Stacey is 9

Download it now, or use the online version.

Release notes are here.

 

On Task Hoarding and Todo Bankruptcy

Last week I watched a British Channel 4 documentary about a chronic hoarder named Richard Wallace. Fascinating stuff.

This was a man who hadn't had a bath in years because his bath, like everything else in his house, was covered to the ceiling in piles of collected junk. To get from one room to the next he would have to swim over the top of his junk pile, ducking under the top of the doorway. And yet he was a fairly normal guy. He wasn't like some angry Smaug, fighting to protect his golden hoard. He had a well developed sense of humour, and you could carry out a normal conversation with the guy.

Yet he had a profound lack of insight. Despite his incredible existence (he slept in a chair, his bed was covered in ceiling-high junk), he didn't see see that he had a psychological syndrome. Pay attention to this, because it's kind of the point: He didn't believe he had an excessive hoarding habit, he felt his real problem was a shortage of storage.

I think what scared me was that he was a little like me. Or a little like you. So it got me thinking about my own hoarding tendencies.

It's normal to do some amount of collecting. I've always done a little here and there. I have boxes and boxes of books under the house. I have a record collection somewhere. I have a jar of buttons (more on that later). But all of that is under control. (After watching the documentary I immediately threw away a box of msdn magazines that I've been holding onto for too long.)

Instead, there is a completely different form of hoarding I engage in, that has gotten out of control.

The thing I collect is incomplete projects. Unfinished work.

My list of incomplete projects is my own personal crazy hoard.

To get from one task to the next, I need to swim over the ceiling-high pile of incomplete tasks, stacked up from every stray project idea I've ever had.

Like a crazy hoarder I mistake the root cause of my growing mountain of incomplete work. The hoarder thinks he has a storage problem (when he really has a 'throwing things away problem'). I say I am 'time poor' as if the problem is that poor me is given only 24 hours in a day. It's more accurate to say... what exactly? It seems crazy for a crazy person to use his own crazy reasoning to diagnose his own crazy condition. Maybe I too easily add new projects to my list, or I am too reluctant to exit from unsuccessful projects. Perhaps I am too reluctant to let a task go, to ship what I've done. They're never perfect, never good enough.

And I know I'm not alone in making the easy claim that I am 'time poor'. So many people claim to be time poor, when really we are poor at prioritizing, or poor at decisiveness, or don't know how to say 'no' (...to other people, to our own ideas).

If only I had a hidden store of time, or if only I had magical organisation tools, or if only I could improve my productive throughput, then, only then would I be able to get things done, to consolidate the growing backlogs and todo lists into one clear line of work, and plough through it like an arctic ice breaker carving its way through a sheet of ice.

So I have to declare todo bankruptcy. Throw everything out and start again.

I've gone back to the trello boards and slashed and pruned and archived and revised and reordered until the whole todo landscape started to make sense. I've dragged in items from other sources (from moleskine's and from starred items in gmail, from 'TODO' comments in my code, from various TODO.txt files buried on my hard-drive... from all over.)

First up, a lot of projects have moved from the maybe pile to the 'never no way' pile.

Example of projects that are officially cancelled as of this moment are:

Not To Do

  1. 'Cop Dog Buddies' -- movie script about two mismatched police dogs thrown together to hunt down their masters' killers.
  2. 'Registry on Rails' -- the web development framework from hell.
  3. Orable -- (pronounced "'orrible") my poor man's oracle IDE
  4. OCD-targeted one-page website "have i left the iron on?"
  5. A life-sized full-body tattoo of my own body, all over my body.
  6. My pro wrestling career
  7. My rap battle ambitions
  8. A range of Pinal Dave T-Shirts
  9. Twee -- a micro-blogging service where posts can only be three words long and must be valid dictionary words in your language.
  10. 'Vampurr' trilogy -- epic love-triangle between goth-girl tattooist, ear-stretching vampire and shape-shifting battle-cat.
  11. Word as a database -- it's Word. As a database.
  12. Operation Butter-knife
  13. Choosing the right fonts
  14. Operation 'Taser my enemies in the face'
  15. Boss monsters feature in unreleased malware easter egg
  16. 'Silmarillion with sock puppets' -- treatment for a feature film
  17. 'Top Gut' -- movie script about the fat camp for the fattest of the fat

Okay, that's a slightly modified version of the real list. The real list is a lot longer and a lot sadder. But the actual thing I'll actually do next is:

To Do

1. Put out NimbleText 1.6.

People have been nagging me about this for a while, and it's pretty much ready, if I can just stop distracting myself with all the other craziness.

Stay tuned.

 

Developer UI Done Right: Mercurial Commandline!

Developer tools are famous for having bad usability. In Code Complete Steve McConnell listed this as a case of 'the cobbler's children go barefoot'. (A fancier term for the same effect is 'vocational irony.' you know, like House MD).

But here's a case where User Experience in a very very developer-specific tool is surprisingly polished, in a way that you wouldn't quite expect.

Follow along if you will...

I'm using mercurial and I decide my work is ready to be pushed. At this point, the stack of tasks in my head (the things I'm trying to achieve) just says "push."

I try to push.

C:...>hg push
pushing to ...
searching for changes
abort: push creates new remote head a751ec334b3e!
(you should pull and merge or use push -f to force)

Interesting! It failed, but not in the typical 'pages and pages of angry red error messages' way. Instead I get a one line error description and then this delicious hint -- I can use 'pull and merge' to get past the problem.

(Note that I've elided any personal details, and hashes have been modified to protect the innocent)

So now the stack in my head says "pull, merge, push".

So I pull.

C:...>hg pull
pulling from ...
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 1 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)

This succeeded nicely. But again there is a hint. Having performed the pull, it looks like I need to merge.

The hint reinforces what the previous command told me and helps me to confidently move in the direction I was already headed. (Or: it serves to remind me that I need to merge, in case I had forgotten). The stack in my head now says "merge, push."

I attempt the merge.

C:...>hg merge
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)

It tells me the merge succeeded. This is pretty spectacular all by itself, but for users of distributed version control systems it's a common enough occurrence, so I'll skip over that bit. Again, I get a neat little hint. It reminds me to commit before I do anything else. So the stack in my head now says "commit, push"

I attempt to commit...

C:...>hg commit
...

This last command gives me only happy messages, which I've not included as they're of no interest. There is no hint what to do this time, as mercurial does not remember that my ultimate goal here was to push. But I haven't forgotten that simple goal. So I try the last remaining step: I push.

C:...>hg push
pushing to ...
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 1 changes to 1 files

And everything is right in the world.

There's something just so neat about the hand-holding that mercurial performs throughout the exercise. I want to pick it up and cuddle it and rub my nose against its little nose.

(of course frequent mercurial users will know shorter and slightly better ways to achieve this same thing, but these little step by step hints are completely adequate for the occasional user like me.)

Now why was all of this amazing? As I said, it's a developer tool, so that's already a good reason to expect poor UI.

But secondly, it's an open source tool. There is a belief that open source software favours usability less than other things.

Third, the hints that were so helpful are a form of documentation. And open source projects very often get criticised for their lack of documentation. Again we have a nice counter example.

Fourth -- it's a commandline tool! A commandline tool! This is a category of software that is not known for being helpful and forgiving to the beginner or infrequent user. Yet mercurial is demonstrating a way in which it is possible to be kind to such users.

All up: I continue to find it nice on a whole bunch of levels. Hat tip to mercurial.

(Next article in this series: "Mercurial And How To Undo a 'hg -forget *' Command, Or, Mercurial, You Evil Unusable Bastard, Why Hast Thou Forsaken Me? Undo! Undo!")