Kort Explores conundrums of the human condition

I was asked by a high school programming teacher how his students could learn to apply what they'd learned about programming to practical applications. At the core of the teacher's question is the age old conundrum that it is far easier to teach skill with tools than to teach creativity with tools. Nearly anyone can be taught how to use a paint brush, but very few of them can use that brush to create a masterpiece.

I was recently faced with having to learn a new programming language while simultaneously designing the project that it would be used to create. Since the end result of my creative learning experience was a functional piece of software that successfully achieved its design parameters, the development process I went through might contain some interesting insights into the way we learn and use languages, suggest at least one way to make the quantum leap to using language as a tool in a creative process, and give me a bit of a soapbox from which to advocate the way I've come to think software should be designed.

I find the process of writing software to be much like writing fiction. The most important part of the process happens before putting the first prose to paper, while the basic plot line and story elements are worked out. If the plot doesn't grab and hold the reader's interest, even the most elegant prose won't make the story worth reading. A brilliant plot has been known to carry a less than inspired prose style.

Many authors fall into the ego trap of writing the book they wanted to write, and then wondering why no one wants to read it. The same is perhaps doubly true of software development. Most authors have an unsold manuscript or two they've written off as a learning experience. I suspect most programmers have a project or two that seemed like a good idea at the time, but which failed in the real world because the programmer's idea of what users *should* want didn't accurately fit the user's real world needs.

My development process starts with defining the purpose of the software. Sometimes the initial inspiration comes from people who can't find software that can do something they need done. Sometimes it comes from knowledge of how an industry operates, and recognizing an unfilled need for a software solution. Most often it comes by accident or chance. As in many aspects of life, the key is recognizing opportunities when they present themselves.

While a few mass market products hit it big, the vast majority fail to earn back their development and marketing costs. The best potential for small developers is to seek niche markets large enough to justify the cost of developing a product, but too small to interest the mass market software companies. Few mass market products survive longer than a couple years. A good niche market can support a small developer for decades if he responds to user feedback by continually refining his product to more accurately serve its user's needs at a reasonable price.

My experience with the C++ computer programming language was in reverse of the usual progression. Instead of looking for a project to help me learn the language, I was forced to learn the language because it was the only way to build a project my customers had requested. My introduction to C++ came about because my customers wanted to use handheld computers running the Windows CE operating system for on-site data entry. I'd never used C++ before, but it was the only language available for WinCE at the time. My language of choice - PDC Prolog - is a declarative language with little in common with C++. I did some work in Pascal - which at least is a procedural language like standard C - over a decade ago, but Pascal has little in common with the object orientation of C++.

I started out with the advantage of already having a pretty good idea of what I was trying to build, and what hardware and operating system would be involved. However, since I had no experience with the hardware, operating system, and programming language that were required by the project, in many ways I was in the same situation as a student facing his first major project.

Reading about programming and studying examples is the only way to start learning a programming language, but I don't really get to know a language until I try to apply what I "think" I've learned from the books to a real life project. There are few more uncompromising tests of programming skill than compiling a project and seeing how it runs.

As I thought about how best to handle the intended task, I made notes in a text file. At first I wrote random thoughts - the kinds of data that will be involved and possible data structures to hold it, potential problems to overcome, user interface considerations, etc. The notes helped to focus my thoughts, getting progressively more organized and coherent as the file got longer, and I was better able to picture the project's internal architecture in my mind. When I thought I could envision an internal architecture that would accomplish the intended task, I started thinking about how to express that architecture in source code.

This is the point in the process where language intrudes its ugly head. Getting past this point is also something of a catch-22 for novice programmers. In order to start writing code, the programmer must be familiar enough with the language to "think in it" - to be able to see the abstract structure and logic flow in the terms and syntax of the programming language. Knowing a language well enough to "think in it" requires a working familiarity that can only be gained by using it, while using it requires already having a working familiarity.

Bootstrapping language skills takes human infants years before they can even write rudimentary sentences, and decades before even the best of them achieve expert proficiency. Fortunately, this process can be shortened for programmers who need to write a computer program in a new language, but don't have decades to get it done.

I started out by acquiring a C++ development system and several books on C++ programming. I looked for books with widely different approaches. I find reading about the same subject from the perspectives of different authors provides a kind of intellectual binocular vision. Three different perspectives tend to be even better than two.

I read through the books quickly the first time, wanting just to get an initial feel for the language. While it may be possible to swim against the current in a river, those who first learn enough about the river to figure out which way the current flows often find the swimming easier. Each language has its own underlying philosophy, and while it may be possible or even necessary to use the language in other ways, it will generally be easier to do things in the ways that are native to the language.

The C++ language contains more functions than I could ever memorize. My approach to any language is to only try to remember the parts I'm going to use all the time, and learn where to look up the rest as they're needed. As I was going through the books, I was also thinking about the logic flow of the project, trying to fit it within the framework of the language. The process is similar to paging through a dictionary with a story line in mind, trying to get a feel for those words that seem to lend themselves to expressing the story. Those words that seem like they might be useful in turn suggest ways to express the logic flow within the vocabulary and syntax of the language. As my study was led by these suggestions, my feel for how things are most easily and/or reliably done in the language was refined.

I made many false starts in the beginning, but the paths of study that presented themselves became progressively more refined and viable as my knowledge increased. As the process continued, the project description in my text file became more detailed, with particular focus on data structures, and how they related to the user interface and outputs like printed reports.

Getting the data structures right is critical to the ultimate success of the project since they in many ways define the nature of the program built to use them. They're also the hardest thing to change in a future version if flaws become evident. However, data structures are another of programming's catch-22s - you can't define the data structures until you know what data you're going to have to deal with, and you can't find out what will be needed to process the data without first defining the data structures. If nothing else, the more accurate your initial guesses, and the earlier in the process you discover the need for changes, the less rewriting you'll have to do later.

The need to design data structures that can efficiently accommodate all of the related information that will be needed by the various processes in the project is pretty self-evident. Guessing what additional information might be needed in future versions is often more of an art than a science - but a little forethought early in the process can avoid major problems in the long term. The Y2K bug resulted from data structure decisions that seemed reasonable at the time, but cost billions to fix in the long term. Designing effective data structures requires a thorough knowledge of the target user and the direction his industry is headed.

My data structures started out simply as a means of holding the information that I knew would be coming into the program. As the project evolved, I had to repeatedly return to the data structures in order to add the additional variables my slowly increasing understanding of the language indicated would be required by internal processes, and to refine my initial simplistic ideas of how the user's information should be stored. An underlying concern at every stage was to incorporate as much flexibility as possible. Data fields that end up never getting used are only a minor inefficiency, while coming up short of places to put necessary information is a disaster.

Once I thought I'd figured out how to accommodate the user's information in digital form, I began a generally top down development that started with trying to figure out the minimum interaction with the user needed to accomplish the intended task. Human labor is very expensive relative to machine labor. Any part of the task that could be logically defined became a potential target for handling inside the software. As a general rule, the greater the percentage of the task I can program the computer to handle "behind the screen", the greater the value of my product to my customers.

My second criteria for the user interface was to make those user interactions that are unavoidable as simple, obvious, and ergonomic as possible. Complex user interfaces may be visually impressive, and might even help sell a few copies to unsuspecting one-time customers. But I've learned the hard way that in the long term, complexity causes confusion and increases the learning curve for my users - which directly increases my tech support overhead. More importantly, both of these factors also contribute to the real costs of using my products. It's the real costs of using my products that ultimately determine if my customers continue using those products, which in turn determines if I stay in business.

An ergonomic interface minimizes the amount of hand, head, and eye movement needed to operate the software. Ergonomics can have a substantial impact on productivity. For example, while the typical pattern of using the mouse cursor to interact with graphic elements on the screen lends itself to such applications as desktop publishing, it becomes counter productive for such uses as entering data from a sheet of paper into a typical data entry screen.

Using a mouse to navigate the screen, and select from menus and check boxes, requires the user's hand be on the mouse, and his eyes on the screen. However, the data he's working on will likely require that his eyes be shifted to the paper, and entering it will most likely require he move his hands to the keyboard. This constant shifting between screen and paper, mouse and keyboard, can introduce so much operating overhead into the process that manipulating the interface takes longer than the actual work of entering the data. This is not a good selling point for business software where productivity directly translates into increased profits and higher wages.

As my random notes evolved into more detailed descriptions and pseudo code, the books became primarily a means of confirming or disproving that there was a way to implement the functions I envisioned using in my written descriptions. When the explanations in the books weren't enough, I used "Find" to search for the function in the example directories to see how it was used. I copied the useful code snippets I found during this stage into a holding file so I wouldn't forget them when it came time to actually start writing code.

Whenever possible I tried to identify those key areas in my evolving vision of the project where my limited and probably flawed understanding of the language created the highest potential for distracting wrong turns and excursions down blind alleys. I build several small test programs to investigate the more critical and/or uncertain aspects of my project - written so that the core of the experiment could eventually be pasted into the main project. Building these small specialized test programs allowed me to experiment on narrowly defined ideas without the overhead and complications of the larger project.

It sometimes took a significant amount of experimentation and run-time tracing of the test programs to come to terms with what was really happening, and revise what I thought I'd learned from the books. But once I had what I thought would be the lynchpin functions of the project working in test programs, I also had the confidence to start writing code.

I continued the trial and error refinement of my still superficial knowledge of the language as I built the project. I focused my initial efforts on building a run-able skeleton program with little more than the required start-up and shutdown processes. I then tried to build the project in stages, with each stage being the smallest part of the task that could be completed and run-tested before adding the next part. Like most "best laid plans", I still had to make major changes in the project as I encountered unexpected complications and/or flaws in my working knowledge of the language. Of course, the earlier I encountered these flaws and complications, the less code I had to go back and rewrite.

As I worked through the easy parts, my fluency in the language improved, as did the pace of development. By the time I ran out of easy parts I was far better prepared to deal with the harder parts. An additional benefit of finishing each small part of the project, and then testing that code, was that whenever I encountered the same problem in later parts, I could just copy the proven correct code instead of having to try to memorize the details. Even if I had to change all of the variable names, just being able to reuse the proven pattern - with all the persnickety punctuation in the right places - was faster and much more reliable than trying to write every line from scratch with no typos.

The single hardest part of programming is getting started on a new project. There are endless reasons not to start. There is of course the ever present fear of the unknown - who knows what traps and pitfalls await the unwary in the dark unexplored corners of a preliminary project description. I'm also not anxious to waste my creative efforts writing code that ends up being abandoned because it was written just before the epiphany that changes everything. Even if I get past my other justifications for delay, it can simply be difficult to find a place to start when viewing the whole convoluted complexity of the proposed project as a monolithic whole.

I find the key is to just pick a small part of the project and plunge in. Even if everything I do at the beginning has to be scrapped later, at least I've gotten past the terrors of the blank page. Once the page is "dirty", it gets much easier to keep writing, with each new addition leveraged off what has already accumulated. I have to overcome the same "crossing the threshold" reluctance when starting a new project description as I face later when starting to write the code. Unfortunately, it's awfully hard to finish without going through the ordeal of starting, so I eventually resort to whatever tricks or ploys will kickstart the process.

The WinCE program that resulted from the C++ learning experience I've recounted above is today doing its job for my customers in the everyday rough and tumble real world of inspectors. The project has gone through several versions in response to the experiences of the beta testers, and to fix a couple minor "behavior problems" that got past my debugging and testing. The general release version of the software has been in daily use for over a year now without generating much tech support traffic - which in my business is as good a stamp of approval as there comes. (I did miscalculate on the type of hardware I expected to be most popular, but the software will also run on the hardware my customers have found better suited to the way they prefer to use the software.)

Best way to learn how to use a language is to use it. My learning experiences haven't all resulted in successful products, but they all contributed to my success as a programmer. The skills I learned creating my failures are what made it possible for me to create my successes.