Architecting, Writing, and Maintaining Large Programs in Visual Basic

Copyright © 2000 by Greg Reddick

Software Failure

This paper was first presented at the DevConnections 2000 conference.

Studies have shown that somewhere between 15% and 40% of large software projects fail. The larger the project, the more likely it is to fail. By failure, I mean they are cancelled or postponed, or are never used. DeMarco and Lister in Peopleware found that in over 500 projects they studied, 15% of them failed. Of projects larger than 25 person-years, more than 25% of them failed. Other studies have shown much higher percentages. Entire books have been written about some of these monumental software failures, discussing such projects as the baggage handling system at Denver International Airport. There are many reasons for these failures. These numbers are appalling. This paper addresses technical issues revolving around the design of programs in Visual Basic to address design methodologies that will give a large project a better percentage chance of succeeding. I'll leave it to others to discuss how to solve management and other issues that contribute to the failure of projects.

Just to give you a firsthand example: I worked on one such failure when I worked at Microsoft. I was a Software Design Engineer at Microsoft from 1988 until 1992, working on what became Microsoft Access. However, the first two years, I worked on a Project code-named Omega. Unless you've seen one of the demos of it that Tod Neilson of Microsoft shows from time to time, you've probably never heard of Omega. It was Microsoft's first attempt at a Windows database program. I would guess that somewhere around 200 person-years were discarded when the project was cancelled. I won't list all the reasons for it's demise and the subsequent revival as Microsoft Access, but part of the problem was technical: The program was just too big. If we applied the techniques that I describe in this paper to that project, it might have had a better chance of succeeding. The techniques I'll discuss use the tools supplied by Visual Basic to avoid the technical pitfalls of many software projects.

Monolithic Programs

Monolithic programs are asking for a software project failure. To be more precise, I should say large monolithic programs are asking for a software project failure. In Visual Basic terms, a project built from a single Standard EXE is a monolithic program. A monolithic program has these flaws:

• Totally unrelated code coexists in the same EXE file.
• Any change to any part of the code requires recompiling and re-releasing the entire program.
• Any code throughout the project may access any form, class modules, public procedure, or public variable resulting in spaghetti code.
• The code is difficult to reuse at design time.
• The code is impossible to reuse at runtime, except through the "shell" function.
• No person on the programming team can get a handle on all the code.

In Visual Basic, you can reuse code at design time by including the same form, class module, or code module into multiple projects (using Project Add File from the menu), but you must be very careful that any change to the shared code does not break any of the other projects.

As the amount of code becomes large, the project can become entirely unmaintainable. I define large as the point where the programmers on a team cannot remember every line of the code they have written and how they interact. For small programs, remembering is easy.

If every procedure in program can interact with every other procedure, and you have ten procedures, then you have 100 different ways they can call each other. If you have 100 procedures, they can call each other 10,000 different ways. If you have 1000 procedures, they can call each other 1,000,000 different ways. I can probably remember 100 things if I really try, but I can't remember 10,000 things, much less a million. When you get to that point, there is a rule of programming:

The day you write the code, you and God knows what it does. A week later, only God knows.

Most of the large software projects that have failed (including Omega) were written as monolithic programs.

Program Quality

Bertrand Meyer, in his book Object-Oriented Software Construction, listed these factors to identify the quality of a program:

• Correctness: The ability of software products to exactly perform their tasks, as defined by the requirements and specifications
• Robustness: The ability of the software systems to function even in abnormal cases
• Extendibility: The ease with which software products may be adapted to changes of specifications
• Reusability: The ability of software products to be reused, in whole or in part, for new applications
• Compatibility: The ease with which software products may be combined with others
• Efficiency: The good use of hardware resources, such as processors, internal and external memories, communications devices
• Portability: The ease with which software products may be transferred to various hardware and software environments
• Verifiability: The ease of preparing acceptance procedures
• Integrity: The ability of software systems to protect their various components against unauthorized access and modification
• Ease of Use: The ease of learning how to use software systems, operating them, preparing input data, interpreting results, recovering from usage errors

I am looking at how to get quality out of a Visual Basic program by addressing these various issues.

Visual Basic Tools

Visual Basic provides several different types of projects that you can create. I am going to concentrate on three of them as one way of approaching building large programs:

• Standard EXE projects
• ActiveX Control projects
• ActiveX DLL projects

A standard EXE is used for building a user interface that will stand alone on the Windows desktop. ActiveX controls also build user interface components, but they cannot stand on their own--they must reside within some other container, such as a standard EXE. ActiveX DLLs are libraries of code that (usually) have no user interface, and are called from other code. That calling code may be in other ActiveX DLLs, ActiveX controls, or standard EXEs.

You use these project types together to build programs. The goal is to build small programs that work, and then tie them together. Presumably, if the small programs work, when you tie them together, you get large programs that work. That presumption isn't necessarily so, but you have a much better chance of making the large programs work than if you just build monolithic programs.

An Example: The Maya Calendar

Let's look at an example from one of my current projects. About ten years ago, I became interested in the Maya. You know, the guys who built these things on the Yucatan peninsula of Mexico (figure 1)...

Figure 1: Temple of the Magician, Uxmal, Mexico

(Photo: Copyright © 1998 Greg Reddick)

The Maya had one of the five independently developed written languages in the world, as well as their own calendar system. Their calendar system could distinguish any date from any other date in history, just like ours does. In fact, it is almost as complicated as ours is. If you don't think ours is complicated, just answer this question: what day of the week does your birthday fall on in 2012?

The Maya wrote dates for things on stone monuments, books, carvings, pottery, walls, etc. In the last 30 or so years, the writing system of the Maya has been cracked open so that we can read most of what they wrote. Frequently they'd say something such as "On this date, we went over to that city over there, beat on some heads, took their king captive, brought him back, tortured him for a while, then executed him." You know, History!

Anyway, back in the Visual Basic 3.0 timeframe, I wrote a program that converts dates from their calendar system into ours and vice versa. I initially wrote it just for my convenience, but I kept making it cooler as I thought of things I could do. Eventually it became a shareware program. It looks like figure 2.

Figure 2: Maya Calendar Program Version 2.02

It is currently showing the Maya date 9.17.0.0.0 13 Ahaw 18 Kumk'u.

That version of the program took second place in user interface design in the Visual Basic Programmer's Journal programming contest in 1995. I've wanted to update it for some time to add some features, but I wrote version 2.0 as a monolithic program (because that's all we had in Visual Basic 3.0) and the changes I had in mind were rather extensive. It wasn't going to be easy. So the question became, how to rewrite it not only to be maintainable, but as a showcase of proper programming technique in Visual Basic so that I could write papers like this one! Not only did I want to write it using proper user interface design, but also using good architecture.

A screen shot from the new version's user interface looks like figure 3 as of the day that I'm writing this. (It is subject to change some before I ship). The program can be downloaded from the Xoc Software web site, http://www.xoc.net, when it is available.

Figure 3: Maya Calendar Version 3.0

Maya Calendar Architecture

Version 3.0 of the Maya Calendar is a rewrite from the ground up using Object Oriented design concepts. I wrote it entirely in Visual Basic 6.0, using a standard EXE, a set of ActiveX controls, and several ActiveX DLLs. It is designed to attempt to be able say that it is Correct, Robust, Extensible, Reusable, Efficient, Portable (to some extent), Verifiable, have Integrity, and be Easy to Use.

To build the interface shown in figure 3, there are currently six projects that build these files: XOCMAYACAL.EXE, XOCREGISTER.DLL, XOCMAYACONTROLS.OCX, XOCMAYATABCONTROLS.OCX, XOCMAYABASECONTROLS.OCX, XOCMAYAENGINE.DLL, plus the help file. (I'll talk about a couple of other related projects later in this paper.) These projects communicated with each other using the architecture shown in figure 4.

Figure 4: Block diagram of Maya Calendar 3.0

These components do the following:

XOCMAYACAL.EXE:

• Presents the window for the user interface
• Talks to XOCREGISTER.DLL
• Places the ActiveX controls from XOCMAYACONTROLS.OCX on the form
• Handles the splash screen and About dialog
• Manages the menu bar

XOCREGISTER.DLL

• Verifies whether the program has been registered
• Displays a dialog that displays registration information, prints an order form, connects to a secure web site for registration, and accepts the unlocking code
• When built as an EXE file, also displays a form that will provide the unlocking code
• Gathers all of its information that is unique to the Maya Calendar program from either properties to the instantiated object, or a file on the disk

XOCMAYACONTROLS.OCX

• Contains two controls: mayaToolbar and mayaTabstrip.
• The mayaToolbar handles displaying the particular Maya calendar toolbar.
• The mayaTabstrip handles displaying the tabstrip and displaying the appropriate constituent control from XOCMAYATABCONTROLS.OCX.

XOCMAYATABCONTROLS.OCX

• Contains seven ActiveX controls for the seven tabs that appear on the tabstrip in XOCMAYACONTROLS.OCX.
• Communicates with XOCMAYABASECONTROLS.OCX.
• Communicates with XOCMAYAENGINE.DLL to get the information to display.

XOCMAYABASECONTROLS.OCX

• Contains two ActiveX controls: mayaExtractor and mayaSpinBox.
• The mayaExtractor control is a modified textbox that automatically parses certain text.
• The mayaSpinBox is a textbox with an integrated UpDown control.

XOCMAYAENGINE.DLL

• Contains an object hierarchy that manages finding appropriate Maya dates for dates in our system, and vice versa, calculating almanacs, parsing date strings, formulating date strings, etc. This does all of the real calculation work.

(Note: In the final program, the three OCX files will probably get combined into one OCX file for shipping purposes, however, from an architectural standpoint they are separate components.)

The Basic Scheme

The idea is to isolate each piece of functionality from the rest of the program. Thus, the XOCMAYAENGINE.DLL component only does calculations. It has no user interface (not even a messagebox, except for debugging). The XOCMAYATABCONTROLS.OCX ActiveX controls only gather and display the information on their individual tabs, getting their information from the XOCMAYAENGINE.DLL component. The XOCMAYACAL.EXE component does almost nothing. It sites the mayaToolbar and mayaTabstrip, handles the menu, displays the splash screen and about dialog, and verifies that the program is registered by talking to XOCREGISTER.DLL. The XOCREGISTER.DLL component is only concerned with registering the program specified--it doesn't even know what program that is.

The reasons for breaking up the program this way are numerous, but it comes back to being Correct, Robust, Extensible, Reusable, Efficient, Portable, Verifiable, Integral, and Usable. Let's visit each of these. In conjunction with the architecture, the program uses naming and coding standards (specifically the RVBA conventions, which can be found at http://www.xoc.net), which helps with maintainability.

Correct

Because each component does only a limited amount of stuff, I can much more easily verify that each component does the task assigned to it. If I can verify the smaller components do what they're supposed to do, it makes it easier to verify that the complete program does what it is supposed to do. Naming and coding standards help here as well.

Robust

This architecture doesn't necessarily help all that much with robustness. However, I have included code to track all unexpected run-time errors that occur when the program runs and logs them. One main point to note--the XOCMAYAENGINE.DLL component doesn't report expected run-time errors. It simply bumps (via Err.Raise) the error up to the component that called it. That component is responsible for reporting or handling the run-time error that occurred. An example is that the Trecena of a Maya data must be in the range from 1 to 13. If the value is outside that range, XOCMAYAENGINE.DLL generates a run-time error. The component that fed the invalid value to the engine is responsible for reporting or handling the error. This means that the engine is entirely separated from the user interface. This will tie into the reusability issue discussed below.

Extensible

Extending each component without affecting any other component is possible. Because I have marked the DLLs and the OCX controls as being "Binary Compatible" in the Project Properties Components tab, I do not have to recompile any other component when I make changes. However, once I release I can only add new properties and methods, not remove or change any existing ones. I will list some planned extensions later. Naming and coding standards help here, because the code is more readable at a later point.

Reusable

This was a prime consideration for this design. As each component is separated from the others, I can reuse those components in different ways. I'll discuss how to re-use the components in an entire section later in this document.

Efficient

This particular design is not necessarily more efficient than a monolithic program. However, it is not terribly inefficient either. Because all of the DLLs and OCX files run in-process, the interaction between the components is fast. In certain circumstances, the fact that the components are separated helps the appearance of efficiency. For example, XOCMAYACAL.EXE doesn't create the controls from XOCMAYACONTROLS.OCX until run-time after it displays the splash screen. This causes the splash screen to appear almost instantaneously rather than waiting until it creates the controls from XOCMAYACONTROLS.OCX. This give the program the appearance of being fast, when in reality it takes the same amount of time to get to where the user can interact with the program.

Portable

This subject ties in with being reusable. Because I wrote the program in Visual Basic, it has certain limitations on where it can be ported. Visual Basic is limited to running in the Windows environments. One major point is that it doesn't run on a Macintosh. This turns out to be an important loss, because many of the potential customers of this program are from academic environments where Macintosh computers are prevalent. On the other hand, in the general world, the vast majority of computers are running Windows. Even on the Macintosh, there is the potential to run Soft-Windows and have the program run from there. One advantage of the particular way that I designed the program is that I can drop the OCX files into web pages and have them run from there, or use Active Server Pages to interact with the engine.

Verifiable

Because each component is small and has a very limited interface, I can test each API to those components with small wrapper test programs. I wrote a test suite that makes sure that I did not break functionality in the XOCMAYAENGINE.DLL component when maintaining it.

Integral

There are two issues of integrity involved with the program: A shareware program has a nag screen that must always appear until the customer has sent me money. When the user sends money, though, it should never appear. This is accomplished by an "access code" that is sent to the user when they register. The XOCREGISTER.DLL component manages the access code. The second issue is use of the OCX files in unauthorized situation, such as redistributing them. I am not particularly concerned about that in this application, but I can place license keys on the OCX files to prevent unauthorized re-use of the components if I wish.

Usable

I designed this program with usability as a prime consideration. (Example: There are almost no dialogs or message boxes in the entire interface--a chosen design goal.) However, for the most part, the architecture has little to do with the interface. One thing that the architecture does promote, though, is the ability to drop different user-interfaces on top of the XOCMAYAENGINE.DLL calculation engine. This means that should the user-interface components fail for any reason, such as being difficult to use, I can re-write the user interface without having to re-write the calculation engine.

Reusability

The two main design criteria from the list above that drove the architecture were reusability and ease of use. It may not seem likely, but there are several uses for a Maya date calculation engine. The interface coming from XOCMAYACAL.EXE is just one interface. Several Maya calendar programs have been written over the years. One such program is a DOS program written by the late Floyd Lounsbury around 1990 that many academics studying the Maya use. This program has a command-line interface and looks like figure 5.

Figure 5: Floyd Lounsbury's Maya Calendar Program

A command-line program has certain advantages over a graphical user interface program if (and only if) you know the syntax for manipulating it. Because Floyd's program has been around for a number of years, and those who use it are partial to it, I decided that rather than competing with it, I would flatter it. Thus the Floyd Emulator interface to the calculation engine. This interface looks like figure 6.

Figure 6: The XocFloydEmulator interface

The command line interface of Floyd's program works the same in the emulator, and the output is virtually the same. Where it differs, the information in my program supercedes the information in Floyd's. This also makes a good test platform for the calculation engine, as the interface is incredibly simple, since it's main job is to parse the command line and pass the results to the engine. Figure 7 shows a block diagram of the architecture of this program.

Figure 7: XocFloydEmulator Block Diagram

Excel provides a third interface to the calculation engine. When analyzing the Maya writings, frequently I need to form tabular listings of dates. Excel obviously doesn't ship with formula for calculating Maya dates, but by writing a little VBA code in Excel, I can call the calculation engine from Excel. Through this functionality, I extend Excel to support Maya date functions. This allows me to construct spreadsheets such as the one shown in Figure 8, part of an analysis I did of one of the Maya codices.

Figure 8: Excel extended to use the XOCMAYAENGINE.DLL functionality

The MayaCR function shown is not an intrinsic part of Excel. It is written in VBA and calls functionality in XOCMAYAENGINE.DLL. The formulas calculate almost every cell of this spreadsheet. Figure 9 shows a block diagram of this functionality.

Figure 9: Excel block diagram

Another possibility, not yet implemented, is to drop the OCX controls onto a web page. The block diagram would look like this figure 10 when this is implemented.

Figure 10: Block diagram of web page

Extensibility

There are a number of extensions planned. One extension deals with the fact that the Maya frequently recorded information about the planets and the Moon when they recorded a date. It is strong confirmation that a correct reading of the date has been made when the astronomy from our models of the universe match what the Maya recorded. I wrote an ActiveX DLL library that takes latitude, longitude, altitude, temperature, atmospheric pressure, date, and time, and produces exactly where the planets and moon should be in the sky at that moment using the best NASA models. By calling this library from the XOCMAYAENGINE.DLL component, I can return that information. Figure 11 shows a partial a diagram that would extend any of the previous block diagrams.

Figure 11: XOCMAYAENGINE.DLL talking to XOCAA.DLL

While the XOCAA.DLL library is mostly completed, I have chosen to defer the integration into XOCMAYAENGINE.DLL until a future release to get this release out the door. The XOCAA.DLL component knows nothing about the Maya. It only deals with astronomy. As such, it could be included into any program that was concerned with the planets such as a telescope auto-location program, or an astrology program, if I were interested in such things.

Another future extension is to report all known recorded dates when the engine refers to a date. The known dates are stored within a database. An example of where this feature is useful: if archaeologists find a new monument with a recorded date, they can immediately tell what other places have recorded the same date. This can give context information that indicates the political connections between the various Maya city-states. The Maya engine would communicate with the database and report the various other occurrences of the same date. Figure 12 shows a block diagram. The database format is not yet determined, but XML is one possibility, allowing web pages to use the same data without intervening OCX controls.

Figure 12: XOCMAYAENGINE.DLL talking to XOCBASE.XML

I was approached about doing a computer display for a museum. For this purpose, I would create an extremely simple interface over the XOCMAYAENGINE.DLL to only allow users to punch in information such as their birthday and have the program calculate the information. In this use, most of the functionality of the engine would be ignored and only the most basic conversions done. I figure that it should take a day or two at most to implement the program. Figure 13 shows a block diagram of this interface.

Figure 13: XOCMAYABASIC.EXE interface to XOCMAYAENGINE.DLL

Because of the design of the architecture, enhancing the program in these ways should be relatively easy.

Applying to Your Programs

Some general rules for decomposing programs into different projects:

• The Standard EXE shouldn't do much. The real work should be done elsewhere.
• Perform calculations in ActiveX DLLs, not in Standard EXEs or ActiveX controls.
• ActiveX controls should form the majority of the user interface.
• Separate by functionality. Dedicate each component to doing one thing.
• Plan for re-use.
• Perform bottom-up programming. Start with programming the ActiveX DLLs, then put the interface on top of them. Write the Standard EXE last.

Some Sample Code

Included with this paper are several projects composing some sample code. Because the Maya Calendar program is complex, and because I don't want to distribute the source code, I have created a set of sample projects that has almost the same architecture, but is much simpler. The architecture looks like figure 14.

Figure 14: Block diagram of XYZMath

When run, the interface looks like figure 15.

Figure 15: XYZMATHUI.EXE Interface

The XYZMATHUI.EXE has only code for creating the tabstrip control, handling the menus, and displaying the splash screen and about dialog. It does none of the functionality of the program. The XYZMATHUI.EXE Standard EXE talks to XYZMATHCONTROLS.OCX. Within this OCX file are three UserControls:

mathTabstrip: Contains constituent controls of a tabstrip control (from the Windows Common Controls), plus copies of mathCircumference and mathArea.

mathCircumference: Contains several controls to gather and display information about the circumference of a circle.

mathArea: Contains several controls to gather and display information about the area of a circle. Figure 15 shows this UserControl within the tabstrip.

The three UserControls talk to XYZMATHENGINE.DLL, which does the calculations for generating the circumference and area of the circle.

The three UserControls from XYZMATHCONTROLS.OCX can also be placed within a web page. Either mathCircumference or mathArea can individually be placed within the page, or the mathTabstrip can be placed to get the tabs to move to either of the two tabs. Figure 16 shows the mathTabstrip placed within a web page.

Figure 16: mathTabstrip running within Internet Explorer

Summary

By breaking up a program into a series of projects, you can build large programs that have a better chance of succeeding. Visual Basic makes that breaking up of the program relatively easy. I showed a practical example, plus a simpler example from which you are free to take apart and steal ideas.

Bibliography

DeMarco, Tom and Lister, Timothy, Peopleware: Productive Projects and Teams, Dorset House Publishing, 1987. ISBN 0-932633-05-6.

Meyer, Bertrand, Object-oriented Software Construction, Prentice Hall, 1988. ISBN 0-13-629049-3.