Kort Explores conundrums of the human condition

Developing computer software involves as much philosophy as programming. My latest experiment in digital dynamics has taken on aspects that I find philosophically intriguing. I'm hoping I can express my abstract observations in terms that will convey at least some of the mental amusements I've enjoyed, to those who aren't as technology obsessed.

I've been slogging away at the details of my latest project for a long time. By necessity, my attention has been largely focused on the individual predicates and clauses I was writing at the time. When I finally got the details working, I was at last able to step back a bit and view the overall system with a bit of perspective. I was more than a little surprised at what I found.

It's been observed that long exposure tends to increase similarities. While perhaps most commonly mentioned by comedians, shared characteristics tend to develop in couples that have spend long lives together. The same tendency has been observed with pets and their owners. So I probably shouldn't have been overly surprised to discover my latest project exhibiting at least faint echoes of the mental structure of its author.

Like most software developers, I've built a variety of largely isolated projects over the years. The internal structures and perspectives of these projects have tended to be as divergent as the tasks they performed. My latest project evolved as a convergence of several originally unrelated lines of development, and is easily the most logically complex and challenging system I've ever attempted.

On some levels I've been working on this project for over a decade. I've been developing my company's flagship product (The Inspection Network) as a desktop application since 1988, adapting it run on at least five different operating systems over the years. Figuring out a way for my products to interact with users through web browsers promised to eliminate the need to waste so much valuable development effort coping with each new round of manipulative abuses in the operating system market. With a web-served user interface, my software would always be able to run on robust and reliable Linux servers regardless of how badly designed the operating system on the user's computer might be.

One of the things I've learned in my years of software development is that projects always turn out to be more complex and difficult than originally expected. I built four increasing complex web-served projects before I had the confidence to try a web-served version of The Inspection Network, and still found the project a daunting challenge.

It was obvious from the beginning that the most difficult aspect of adapting the existing desktop software was going to be the data entry user interface. Dealing with the vagueries and inconsistencies of human users is much harder than any of the tasks the software can do on its own.

My intended objective was to provide a simple easy to use web-served user interface for entering inspection data, as well as dynamically changing the structure and content of the underlying inspection "format". Combining simplicity and flexibility turns out to be a very complex problem. Simply serving up a fixed sequence of predefined screens would have been much too easy. Making the content of the screens, as well as their sequence, dynamically reconfigurable provided a much more "interesting" challenge.

The rippling ramifications of making the structure and content dynamically reconfigurable rapidly expanded into a dizzying assortment of interrelated factors and supporting complexity. Coping with the uncertainty of an Internet socket connection inserted between the user interface and the hardware on which the software is running, contributes additional levels of "excessively interesting" interdependent complexity.

The project had already accumulated a "non-trivial" degree of complexity even before starting this latest effort. The internal structure of the system has gone through four major evolutionary redesigns and countless minor releases over the fourteen years it's been in use. Over time, two major processing areas (engines) have evolved to handle the interrelated issues of system state and process stage. These increasingly specialized engines have been subject to substantially different influences during this evolutionary process. They also brought substantial amounts of inherited baggage from their very different ancestry into the dynamic, since each new incarnation included substantial amounts of code adapted from previous generations.

The evolution of the original core task engine has been primarily influenced by the demands of the primary task of the software - the complex multi-stage process of turning a mixture of raw inspection data, machine generated analysis data, and additional human provided data, into a professional quality natural language (human readable) narrative report. The task engine operates within an abstract environment that is defined by the interactions between the staccato flow of user supplied meta-data, and the rules, conditions, and criteria of its various knowledgebases and internal databases.

The interface engine evolved in response to the increasing complexity of interacting with a human-centric event driven outside world. The environment in which it evolved is dominated by hard data and real time events. In addition to the basic input/output functions required to manage its I/O port, the engine must interpret incoming events occurring on its external interface into meta-data that can be used by the task engine, and interpret the outgoing abstract meta-data returned by the task engine into symbolic forms that can be understood by external machines and humans.

As these engines have evolved, they've grown increasingly isolated from each other. But while they operate in very different environments, they have a continuous need to exchange information and influence each other's operations.

The interface engine knows how to parse incoming HTML pages and assemble outgoing pages. It has a repertoire of various basic means of expression - control buttons, data fields, menus, text areas, etc. An external knowledgebase provides it with rules and configuration data for assembling these basic components into 22 types of screens - text editor, multiple choice, selection menus, control menus, filename dialog, etc.

However, most of these screen types must still be configured to serve a specific purpose in order to serve the needs of the task engine. For example, the interface engine knows how to build a generic editor screen, while the task engine has to deal with dozens of different kinds of text information that at some point in the process might have a need for an editor screen user interface.

The interface engine can do a lot of things, but it lacks a purpose for its capabilities - and content for its screens - without the task engine. The task engine needs to request information from the user in an orderly structured manner that is inherently incompatible with the event driven nature of a web browser user interface, and depends on the interface engine to take care of all the messy details. In practice, the task engine provides the interface engine with meaning and purpose for its interactions with the outside world. The interface engine in turn provides the task engine with a means of satisfying its obsessive craving for information, and symbolically expressing its abstractions in the outside world.

The interface engine quantifies the events of its world as transactions. It starts a new transaction record when it detects an incoming socket connection on its I/O port. It then processes the transaction through a sequence of steps that progressively "fill in the blanks" in the transaction record. Once it knows enough about the event, it builds an outgoing HTML page response and closes the transaction.

The interface engine doesn't understand the data it finds in events. It passes any data to the task engine for interpretation. At the logical half-way point in dealing with a transaction, the interface engine consults a specialized embedded expert system for instructions on what to do next. This embedded expert system tests information from both engines against a knowledgebase of over 150 rules to determine the appropriate next system state and process stage. This "third party" mediation coordinates the activities of the engines, ensuring that the interface engine builds the right type of screen for the meta-data supplied by the task engine.

The focus of the task engine is the acquisition and interpretation of abstract data within the context of an inspection format. If every information request it passes to the interface engine returns the requested information, the task engine will simply work through the list of checklist modules it finds in the inspection format, pulling up each module in turn and stepping through whatever sequence of information requests/questions it contains. Baring unexpected user actions and/or other complications, saving the information it has accumulated completes the task from the perspective of the data entry task engine.

One of the most interesting aspects in the evolution of this project is how the perspectives of the interface and task engines have evolved in different directions. The interface engine has become reactive due to its focus on responding to short term external events. The task engine has become assertively goal oriented due to its focus on achieving a (relatively) long term goal - but prepared to deal with distractions and detours along the way. The interface engine views each momentary event as a discrete transaction with a clearly defined beginning and end. The task engine sees each event as just one of many stages in an ongoing process that "ends when it gets to the end".

The constraints of having to protect against a potentially malicious or otherwise suspect socket connection requires the interface engine to operate within a structured environment that only accommodates a limited number of acceptable logic paths, filters anything "unusual" out of the incoming data, and rejects out of hand any events that deviate from its limited range of expectations.

The task engine's logical environment is dynamically defined by a variety of factors. While its focus during each event is directed and logically purposeful within the context of its current environment, its environment is changed by each event - sometimes subtly, sometimes profoundly. It must continually adjust its perspectives to accommodate new information, "on the fly" changes to the logical structure of the inspection by the user, data errors, and other deviations from its intended logic path. While it continually seeks to progress toward its long term goal, the goal it seeks may well be changed by the events it experiences along the way.

Of course, all the commotion behind the screen is just the means to achieve a larger objective, not the objective in and of itself. The activities of the interface and task engines must ultimately complement each other in their shared overall purpose of making life easier for the user. After all, my motivation for making my software more intelligent is so that it can do more for my customers, not do more to them.

Computers ought to be a lot smarter than they are today. The processing power of the hardware has increased exponentially over the past decade, but there has been precious little progress in making computers easier to use. To the contrary, most software companies seem to be obsessed with making their products ever more densely packed with flash and dazzle clutter that looks impressive on a spec sheet, but in the real world of the user makes accomplishing even basic tasks ever more complex and confusing.

Software delivers its greatest value when it uses the complexity on the machine side of the screen to reduce the complexity on the user side. Computers will only achieve their potential when they start using their processing power to think for the user instead of wasting their cycles filling his screens with ever more distractions for him to think about.

Producing a professional quality inspection report shouldn't take more technical knowledge than performing the inspection it describes.