On law and order last night they demonstrated the Infinite Scalability Principle of Surveillance Cameras on Television Shows.

They used a computer to zoom in on a photograph taken for a gymnasium identification card, and they found a figure lurking outside the gym! They then zoomed right in on this figure until they had a crystal clear image of the lurker (who did indeed turn out to be the murderer!)

That must be a pretty expensive camera they use for ID's in that gymnasium, what with having near-infinite resolution. Most identification card photographs just take grainy, warped ugly little photos. (Or is that what i really look like?) I bet their membership fees are near infinite too.

(continues... more about tech-mistakes on tv)

This is a common tech-mistake on tv. Surveillance cameras are the most frequent culprits:

 
	      	CHIEF
	--Let's scan through six hundred hours of 
	surveillance camera footage, taken from thirty 
	different cameras, to see if we can find anything 
	to corroborate our witnesses testimony. 
	
	(Three seconds go by.)
	
  	     	OFFICER
	--Dammit chief we found nothing.
	
    	  	CHIEF
	--I have an idea. Why don't we all stand around 
	talking for a moment, but leave one of the video 
	tapes still running on one monitor behind us.
	
	(They talk amongst themselves for three seconds.)
		
    	  	OFFICER
	--Hey! will you look at that? There's a car in 
	the background that matches the rough description 
	we got from the eye witness. Let's zoom in 
	on the number plate....
	
	(They zoom in, possibly using the help of 
	the office geek, who is fat, stutters alot 
	and has poor personal hygiene. The geek may use
	some magical 'image enhancement' filter which
	interpolates everything imaginable to create
	a perfect image.)
		
	Although the car was photographed from a 
	distance of 300 hundred yards, in a dark 
	carpark, at midnight, by a grainy  
	black and white surveillance camera, 
	we see a remarkably clear image of the 
	license plate:
	
	555ZTE4WK1 - Washington.

 	
    	  	CHIEF
	--Let's run that plate!

	

IN order to search the database, they will invariably need to perform what I call a "Visual Search."

A Visual Search is where every record in the database scrolls over the screen while it is being compared. This is so spectacularly inefficient it could only have been invented by the television and movie industry.

Let's investiage the metrics of 'Visual Search'. I've done a quick simulation using a console application, and to get the sort of scrolling you usually see on these shows, you need to scroll about 40 records a second, no quicker. At a wild guess I'd say there are 300 million vehicles in the United States. On average you'd expect to find a match by scrolling through half of those, about 150 million records.

(Leon loads calc.exe and does a lot of diving by 60, 24, etc.)

That's about three months of scrolling. But very rarely do you see a little caption that says "Three Months Later".

Also, if you consider that you're transmitting every record of a nation-wide database to a client terminal, there's gonna be 'round-tripping' delays like you wouldn't believe.

You've got to expect to see the occasional scene like this:

 The chief leaps out of his chair:
	      	CHIEF
	--Holy fricking jesus! The departmental 
	internet bill is six bazillion dollars!
	
	(he calls in the fat, ill-kempt stuttering geek)
	
	      	GEEK	
	--It's all these 'Visual Searches', Chief. 
	The script-writer who designed our information 
	retrieval system had no idea about n-tier architecture.
	
	      	CHIEF	
	--Shut up Geek! You're spittin' baloney at me! 
	And what are all these god-damn lawyer bills!
	
	(in comes the sleek, slimey-haired lawyer 
	with the thin briefcase)
	
	      	LAWYER	
	--We're being sued by some real police officers 
	because we trivialise their work, and represent 
	them as a bunch of stereotypical fools. The 
	royal grammar society have a few issues 
	with you as well.
	
	      	CHIEF	
	--We'll kick their asses! 
	
	      	LAWYER	
	--Sorry Chief. I have to recuse myself, owing 
	to a conflict of inteterst. I've been 
	disbarred for protraying lawyers as sleek, 
	slimey-haired and for having a thin briefcase. 
	Also, my use of the words "recuse", "disbarr" and 
	"conflict of interest" don't actually represent 
	correct legal usage. What's that smell?
	
	      	GEEK	
	--Sorry. Just burped.
	

And that's all I've got to say about television right now.

It is about time someone told you!

YOU ARE NOT INADEQUATE.

• You do NOT have to refactor all your code.
• You do NOT have to keep up with the latest news from microsoft, and know everythnig there is to know about longhorn, whidbey, avalon, XAML, indigo and star wars III.
• You do not have to have perfectly de-coupled tiers in your technology independent SOA software.
• You do not have to comply to every standard, achieve the perfect balance between maintainability and performance. Usability and familiarity.
• You don't have to do "first things first every day"
• You DO NOT have to memorize and understand every patten the gang of four have catalogued.
• You do NOT have to read every technical blog, print out every technical article and learn every technical thing there is to learn.
• You are beautiful just the way you are.
• You are brilliant, interesting, wise and fun to be around.
• You rock.

And you are not inadequate.

Reading too many blogs and technical books can sometimes lead you into a mindset where you think you are so far technically inferior to these 'great minds of redmond' -- but you probably do an excellent and practical job, just as you are.

so keep happy, kiddo.

secretGeek recommends:Feeling Good, by Dr David Burns

If, like me, you eat your lunch at your desk (and I know you do, cause while you're reading this, i too am reading you) then you'll know that once in a while you've just got to pick up that keyboard, turn it upside down and shake out those crumbs. Shakity, shakity shake shake! Shakity, shakity shake shake! Out comes cheese! Out comes bread! Out comes chocolate! Out comes tuna! This is more fun than updating your timesheet! More fun than deleting old emails! The most fun you can have in your cubicle during daylight hours and with your pants on! Shake that keyboard baby!

(no further details provided)

(I said no further details provided, dimwit... but feel free to leave a comment...)

I've just returned from a trip to the Whitsunday Islands on Australia's Great Barrier Reef. I declined to inform you (my small but devoted audience) that I would be away for a week, in the hope that you wouldn't use the opportunity to break into my house and steal all my stuff.

The Whitsunday islands are fascinating. Amongst other facts, I learnt that the Clown Fish (Amphiprion percula) has now, apparently, been renamed 'the Nemo fish'.

(continues...)

A week without a computer is a fascinating experience. I might recommend it to you. Or alternatively, you can do a google search on 'life without a computer', and get all the facts in a fraction of the time.

I'll confess though, that I continued to dream of software. One particularly vivid dream involved the file explorer of a futuristic operating system. Hard-drives were respresented by islands, shortcuts were bridges, folders were highways streets and alleyways (of smaller and smaller sizes, the further into the file heirarchy you flew) and files themselves were represented by buildings: different types of buildings for different types of files. Navigation was achieved by flying over the panoramic isle of files. Ahhh. Too many cocktails. Too much sun.

Now I have to devise a way to become a billionaire by Monday, so i need never work again. Hopefully I've forgotten all my passwords in the meantime...

I'm disturbed by a change in the way I use windows. I strongly prefer keystrokes to menus.

To run a program I press "[Windows]-R" to bring up the "Run Dialog," then I type the name of the Exe I want, e.g. "WinWord", "DevEnv", "Excel", "Calc", "Notepad", etc.

Is this some malingering malady? A form of Linuxitus? Or a deep-seated DOS depravity?

(end.)

When you dream of software regularly, it's a bit of a worry. Anyhow, this is an idea I had while mowing the lawn today. I think it could really speed things up interconnecting legacy systems with the XML-aware world. What do you think?

XPLT borrows from XSLT and Regular Expressions (RE), but also provides a more human-readable style of Pattern-Matching, encapsulated within the Pattern element. Patterns are combined with templates and flow control semantics, that allow for very flexible document transformations.

XPL Transformers could be written on any platform, for any technology. Collections of specific and re-usable XPL documents for converting between any two text-formats could be shared and distributed.

Underlying principles of XPL

1. Human Readable
2. Not concerned with Terseness
3. Platform Independent
4. Transport Indepedent
5. XPL documents are valid XML documents

How is it used?

Here are some typical scenarios for which XPLT is useful.

• EDI documents, containing data in fixed-width fields, need to be converted to XML for transmission to a web service
• A legacy system produces CSV documents. They can be converted to XML using XPLT.
• A trading partner produces XML documents that are not always well-formed. They can be 'pre-processed' with XPLT to produce well-formed XML.
• An RSS aggregator receives some documents that are in an obscure format. They can be transformed to valid RSS using XPLT
• A HTML doc needs to be parsed to retrieve certain values.
• VB.net code needs to be changed into C#
• HTML code includes embedded font tags. It needs to be altered to use CSS.
• Wiki Text needs to be converted to xHTML.
• A regular expression needs to be converted into a human-readable explanation

These problems can be solved by writing custom parsers, or, at best, through clever use of Regular Expressions.

Maintenance of such code can prove remarkably expensive. Why?

• System boundaries are *always* in a state of flux.
• Custom parsing is notoriously buggy -- even if it runs error-free for six months, you know there is a chance that an unexpected character combination could upset it.
• Regular Expressions are deliberately terse. This limits the number of developers who take the time to master them.
• The platform and underlying technology of a system boundary can change. Any platform or technology dependent code then needs to be rewritten completely.

Features of the Language

The main features of the language are:

• the Pattern
• flow control
• the replace construct
• the template

From XSLT it borrows elements used for these constructs:

• If/else statements
• Choose statements
• for-each
• Sort

In XSLT, the 'Match' and 'Select' attributes reference an Xpath expression. In XPLT the equivalent attributes reference a named Pattern element.

The Pattern Element

A pattern element serves a purpose analogous with a Regular Expression (RE). It can, infact, be simply a RE, for example:

<pattern name="email" RE:match="(\w[-._\w]*\w@\w[-._\w]*\w\.\w{2,3})" />

[Thanks to Darren Neimke for the RE example, taken from RegExLib.]

Any such pattern can also be expanded using the far more verbose pattern semantics of XPLT.

<pattern name="email" >
	<all> <-- inside an 'all' element,
       all rules must be applied --/> 
		<chargroup occurs=1>
			<char class='word'/>
		</chargroup >
		<chargroup occursatleast=0>
			<any> <-- inside an 'any' element, 
       only 1 rule must be applied --/>
				<char class='word'/>
				<char value='-' />
				<char value='.' />
				<char value='_' />
			</any>
		</chargroup >
		<chargroup occurs=1>
			<char class='word'/>
		</chargroup >
		<chargroup occurs=1>
			<char value='@'/>
		</chargroup >
		<chargroup occurs=1>
			<char class='word'/>
		</chargroup >
		<chargroup occursatleast=0>
			<any>
				<char class='word'/>
				<char value='-' />
				<char value='.' />
				<char value='_' />
			</any>
		</chargroup >
		<chargroup occurs=1>
			<char class='word'/>
		</chargroup >
		<chargroup occurs=1>
			<char value='.'/>
		</chargroup >
		<chargroup occursatleast=2 occursatmost=3>
			<char class='word'/>
		</chargroup >
	</all>
</pattern>

Okay - the names Char, Chargroup etc, are pretty ugly. But I've just invented them now. The structure of those pieces could be altered and improved, I am sure. Just food for thought. Rather than

	<chargroup occurs=1>
		<char class='word'/>
	</chargroup >

	<char class='word' length='1'/>

And Rather than

	<chargroup occursatleast=0>
		<any> 
			<char class='A'/>
			<char value='B' />
			<char value='C' />
			<char value='D' />
		</any>
	</chargroup >

You could just say:

	<chargroup occursatleast=0>
		<any>ABCD</any>
	</chargroup >

I guess you would also have 'greedy' and/or 'non-greedy' attributes.

Here's the original picture I came up with -- the complete loop converting proprietary text into XML by using XPLT. Then converting the XML back into properietary text, by way of XSLT.

I'm damn tired now -- so i'll stop there. Cheers!

Disambiguation

XPL is not any of the following:

• A language for writing compilers.
• A scripting language for the Note Bene word processor.
• A MIDI programming language for Amiga.
• A language called eXtensible Programming Language.
• A Pascal-like language named XPL0.

Here's a problem you're likely to get when setting up a new dev' box.

Solution

Open a command window and run 'aspnet_regiis' with the -i option

	aspnet_regiis -i

What is the -i option for?

(Continues, if you're real damn interested...)

A quick way to tell if ASP.net is running, (and what version) is to deliberately get a 404 error on a ".aspx" extension. For example, navigate to:

	http://localhost/AnyMadeUpName_MXKAXMAK.aspx

If asp.net is running:

• you'll get a .Net error page (with the version written at the bottom of the page).

If IIS is running:

• you'll get an IIS error page (The Page cannot be found)

If IIS is not running (at this site)

• you'll get an explorer error page (The Web Site cannot be found) [unless some thing else hi-jacks your browser, such as google toolbar.]

A related problem you may find if client-side scripts are misbehaving the whole time... you need to rebuild the client scripts. Here's how:

	aspnet_regiis -c  

What is the -c option for?

Related resource: The VS7 Debugger doesn't work. What can I do?

Why suggest add-ins rather than write them? because as Ratbert says: "I'm more of an ideas rat."

You're free to implement these suggestions if you wish. If you end up winning Roy's competition the prizes are all yours, though I wouldn't mind Mike Gunderloy's  book  (Coder to Developer).

The suggestions fall into two categories:

1. 'Producitivity' Enhancements
2. Embedded mini-apps

There's about thirty suggestions. I make no guarantee that these things have not already been implemented.

'Productivity'^* Enhancements

[* I put quote marks around the word Productivity because, as you'll see, it's a fair stretch of the imagination to think that any of them would actually increase your productivity.]

• Highlight a keyword, press a key, and a google search is performed.
• Pretty much any idea from Roedy Green's *classic* SCID page
• Windows Form designer improvements:
  • Ability to zoom in/zoom out on forms
  • Able to 'draw' onto a form much like in Photoshop/Paintshop Pro/Photodraw et al, and have your actions rendered into GDI+ underneath (and later, XAML)(i wrote a pascal version of this in my youth)
• Spell checking of comments, and embedded strings. With custom dictionaries and Multi-language of course.
• Mixed fonts (font-sizes, font-face etc.) for different situations, e.g. Class Declaration different size and font-face to comments etc.)
• In-line calculator (ie. highlight an expression, hit a key and the answer is spat out on the next line)
• In-line expression evaluator (ie. highlight an expression, hit a key and the return value is spat out on the next line)
• Ability to have persisted bookmarks, and embedded links within comments.
• Some more 'wysiwyg' way of displaying code, or displaying asp.net
• Language conversion macros (e.g. C# <--> Vb.net)
• Annotation/Journaling system that keeps parallel notes associated with code, e.g. date that code was written/modified, other notes you wish to add: notes that are not recorded in comments. Can use this for code reviews etc.
• Of course you should be able to walk through your code as a virtual reality 3-dimensional (or 4-dimensional?) landscape. This is just an absolute must have. All code you've ever written could be in the landscape, or better yet -- all code anyone has ever written. It could be rendered by topic, reliability, preference, readability, all kinds of soft and fuzzy categories, as well as things like purpose, inputs, outputs, assumptions. It's a Piece a piss.
• Zoom in/out on your code -- view the whole project on the screen at once -- calls between sections shown as links etc. You hover over a part and it can inflate up to a readable size.
• Code view: Replace variables with custom icons, shaped according to their type, colored according to their scope and with more specific image attributes based on their meaning (ie, based on their name) (Note: When I first heard the term 'Visual Basic' I imagined that this was what it did!)
• Have "tool tip text" appear in a dedicated little side panel, rather than where you're typing. A history of all tool tip texts is maintained in a listbox. That way they're easier to cut and past for google searches, etc.
• A chess timer for developing: i hit you can start it/stop it -- see how long you're coding for. Track this against tasks etc.
• Refactor "if/else" to "case" statements.
• Ability to embed images/diagrams into the comments/annotation.
• Asp.net: A way to have Buttons for "HTML view | Code View | Code behind" right next to each other where they should have been all along (!!)
• Screenshare: for pair-programming/Xtreme Programming - to allow multiple developers to **really** be on the same screen/workspace at once (more than one mousepointer/keyboard entry/monitor...)
• Wallpaper. 'nuff said.
• Pinvoke auto wrapping.You navigate to the win32 call you wanna make -- it puts a pinvoke call in a wrapper classs (if the wrapper class doesn't exist yet, it creates it).
• When you cut or copy code, it copies as richtext (i think). A nice option (for bloggers) would be 'Copy as HTML'.
• Enable 'block select'
• Undo/Redo List

Embedded mini-apps

Most of these are fairly obvious and I'd guess that most have been done before. I'm not gonna provide links though.

• Project Management integration
• email and or FTP client
• A 'reverse-boss-key' so that if you're coding when you should be gaming, the screen can instantly convert to some plausible looking shoot-em up game
• Mp3 player
• Icon editing
• Chatting
• A plug-in that lets you run X-box games inside a tab within Vis Studio.
• Blog publishing and or reading
• (better) versioning/control integration
• Excel Lite Plug-in

The more I think about these add-ins, the more I realise 'less is more.'

While I love so many of the enhancements in Visual Studio .Net to date, I think that

most of all it could benefit with being a lot more performant.

This might be achieved only by moving to a Longhorn model where GUI rendering is handled via the graphics card rather than bitmap rendering in regular memory.

Maybe the best 'add-in' would be:

a minus-in that lightens the load.

Excel-Lite (XLL) has all the calculating power of regular Excel, but without the bloat, the baggage, the bulge, the.. the.. obligatory overheads, the albatrossicle accoutrements, the embarrasing encumbrances, the chaff.

For generating SQL statements, or HTML, or any sort of code, for manipulating raw data, you can't go past XLL.

There is no limit to the number of rows or columns you can use. No limit to the number of sheets.

XLL does not reformat your data. It won't drop the '0' from the front of a number. It won't alter the format of a field that happens to resemble a date. Currency symbols don't confuse it.

It has excellent support for Regular Expressions. Plus a genuinely comprehensive 'reveal-codes' mode to help you get to grips with your data.

Okay, there's no ability to set text colours or bold or italics or varying fonts or font-sizes, and there are no charting capabilities and even no printing (without plug-ins). But those features are outside the intended use of the product. The trade-off is a fast, versatile, bloat-free application, targeted at programmers and information-workers.

It's quick to load. And damn quick at performing calculations. It takes up about 100k of memory (excluding spreadsheets) and has a versatile undo/redo capability that appears to be unlimited.

Files are internally represented (and stored) as XML. Hence spreadsheet searches can be performed with XPath!

(continues...)

You can load data from most any format ever devised (Yes, VisiCalc too!). And there are plug-ins that allow you to define new custom input formats, too. (What else would you expect?)

You don't have to import from files either -- you can import from webservices, databases, ftp, smtp, streams ... you name it! Files can only be saved as XML (or compressed XML). But the package includes a wealth of XSL-templates, so you can transform your XML output for presentation, or into other formats, if you wish.

Keystroke Macros can be easily recorded, stored, edited or replayed. For more serious functionality extensions, you can write a plug-in in any .net language -- just implement the iXLLPlugIn interface.

There's already a vast wealth of plug-ins available for anything you might want to perform. Plug-ins exist for database access (it can act as a replacement for SQL query analyzer!), for manipulating XML in any way imaginable, for authoring XSLT, CSS, batch files, circuit design, statistical analysis, timetabling, reporting, scheduling, and more fractal-toolkits than you could ever know what to do with. There are some quite impressive plug-ins for integrating with numerous applications, such as Biztalk, Infopath, MS-CRM, Sharepoint and even that lame-elephant, SAP.

Only one real problem. Excel-Lite is not a Microsoft product. In fact, no-one has written it yet. It's just a figment of my imagination, dreamt up a few days ago thanks to a post by Dave Burke. But hopefully, i'll have my hands on it one day not too soooon.

Ahhh... dreaming of software...

Some of the people who read this blog are amongst the cleverest, most productive and influential programmers and technical authors in the world. This, despite the fact that I am an utter fool, obsessed with toilet humour and cheap satire. I am utterly bewildered at how fortunate I am to have the ear of such industry legends as Kathleen Dollard, Julia Lerman and Mike Gunderloy, amongst others.

I was so blown away, just now when I found a comment from Kathleen Dollard that I burnt the meal I was half-way through cooking. The long suffering Mrs. Secret Geek was less than impressed, but ultimately sympathetic.

Kathleen, in case you don't know, is the author of 'Code Generation in Microsoft .NET', a topic which I am right behind, even though I confess to not having seen the book personally. On her site she uses a lord of the rings metaphor to explain the importance of Code Generation:

"...line by line code creation [is] the Ring of Doom we've all been dragging around. In spite of all our brilliant technological advances, we still write code the same way we did 20 years ago - line by line in an editor. The ring is our blindness to recognizing this legacy because we can't see past the need get that next line of code written."

I couldn't agree more. As my boss, Bill Murphy, always says:

'you simply cannot be profitable in this business unless you can automatically generate the basic maintenance programs that we find ourselves writing again and again, year after year.'

The company I work for spends a lot of time getting code generation right. It can be a frustrating purpose, writing or fixing templates when you'd rather be writing or fixing code. But the economics are a no-brainer.

Here's to code generation!