More Memory for DOS Exec Kim Kokkonen As many have lamented, the 640K of memory available to DOS programs is looking smaller every year. With TSR's gobbling up memory on one end, and our applications growing larger on the other, it is easy to use up all the space and then some. Of course, necessity is the mother of invention, so desperate DOS programmers have devised a number of ad hoc methods to cram more functions into the same space -- by using expanded and extended memory, overlays, and so on. This article describes another such method. We've enhanced the DOS Exec function by swapping most of the calling program into expanded memory or to disk, and giving all that free memory to the child process. When the subprocess is complete, the calling program is swapped back into place and continues normally. This technique is especially valuable for menuing environments which must execute other large programs, or modern programming editors which are expected to spawn huge compilations at the touch of a key. In fact, it's useful for any program that must invoke another. The swapping Exec function is implemented in a Turbo Pascal 5.0 unit called ExecSwap. The real meat of the code is written in assembly language, however, and with some changes could be linked into other languages such as C or Fortran. Turbo Pascal Program Organization --------------------------------- To explain how ExecSwap works, we'll need to delve into the organization of a Turbo Pascal program. Let's examine the program shown in Figure 1. What this program (named X) does isn't important. We'll just use it to show the arrangement of memory. X uses two of Turbo's standard units, Crt and Dos. It also implicitly uses the System unit, as does every Turbo Pascal program. Figure 2 maps out the various segments. (You can see a similar map of a real program by having the compiler create a MAP file and inspecting the segment map at the beginning of that file.) It's important to note that each Pascal unit has its own code segment (denoted by CS_xxx in Figure 2), and that the code segments are arranged in what might seem like reverse order. That is, the unit appearing first in the USES statement is linked at the highest memory address, while the main program has the lowest code segment. If the program doesn't need to use the heap, the memory above the heap base may not be allocated. Figure 1: Example Program program X; uses {System,} Dos, Crt; begin ClrScr; Exec('C:\COMMAND.COM', ''); end. Figure 2: Memory Map of Example Program PSP: program segment prefix lower addresses CS_X: X code | CS_Crt: Crt code | CS_Dos: Dos code v CS_System: System code higher addresses DS: initialized data | uninitialized data | SS: stack v HeapOrg: heap base HeapPtr: heap high water mark available heap space FreePtr: free list FreePtr+1000h: top of program available DOS memory xxxx: top of memory ExecSwap's goal is to copy most of the memory used by the program to secondary storage and then to deallocate that memory. ExecSwap needs to leave only enough of itself behind to call DOS Exec and restore the image when the child process returns. By this criterion, the best place for ExecSwap's code would be in the main body of the program. In this way, it could start swapping memory at the lowest possible code segment and free the most memory for the child process. In Figure 2's terms, it would start swapping at code segment CS_X and continue to the top of the program. After deallocating memory, the only overhead would be the program segment prefix (256 bytes) plus the portion of segment CS_X required to undo the swap. Figure 3 shows what memory might look like while the child process was active. The rest of program X would have been stored in EMS memory if available, or in a disk file if not. Figure 3: Memory Map while Child Process is Active PSP: program segment prefix | ExecSwap CS_X: X code (partial) | overhead .-------------------------------------------------- | child program program segment prefix | ... | xxxx: top of memory There's another factor to consider, though. ExecSwap should be convenient to use in more than just one program. Hence, we've made it a self-contained unit which is available just by adding it to the main program's USES statement. Considering Figure 2 again, it's clear that when we USE ExecSwap we want to add it at the very end of the list. In that case, the memory map will look like Figure 4. The memory that remains allocated during the Exec is the PSP, the code in the main program X, and whatever part of ExecSwap must remain resident. Figure 4: Memory Map after using ExecSwap PSP: program segment prefix CS_X: X code CS_ExecSwap: ExecSwap code <----------- CS_Crt: Crt code CS_Dos: Dos code CS_System: System code ... xxxx: top of memory The main program's code segment need not be very large, of course. In the extreme case, the main program would consist of nothing but a USES statement and a single procedure call to another unit. This reduces the overhead of the Exec call to essentially just the PSP plus ExecSwap itself. And that's not much: ExecSwap's resident portion consumes less than 2000 bytes. Using ExecSwap -------------- Before we plunge into the mechanics of ExecSwap, we'll describe how it is used by an application. The unit interfaces three routines, shown in Figure 5. Before performing an Exec call, the program must call InitExecSwap. This routine computes how many bytes to swap and allocates space to store the swapped region. Figure 5: ExecSwap Routines function InitExecSwap(LastToSave : Pointer; SwapFileName : String) : Boolean; {-Initialize for swapping, returning TRUE if successful} function ExecWithSwap(Path, CmdLine : String) : Word; {-DOS Exec supporting swap to EMS or disk} procedure ShutdownExecSwap; {-Deallocate swap area} The swapped region of memory starts just beyond the resident portion of ExecSwap. The programmer must specify the _end_ of the region with the parameter LastToSave, since the choice depends on how the program uses the heap. What we choose for LastToSave affects only the size of the swap file, or the amount of EMS memory needed, but has no effect on resident overhead during the Exec call. There are three reasonable values for LastToSave. Passing the System variable HeapOrg tells ExecSwap not to save any part of the heap; this is the correct option for programs that make no use of the heap. Passing HeapPtr causes ExecSwap to save all allocated portions of the heap. Only the free list is ignored, so this is a good choice for programs that don't fragment the heap. Passing the expression Ptr(Seg(FreePtr^)+$1000, 0) tells ExecSwap to save the entire heap, including the free list. This is the most conservative option, but it may lead to swap files approaching 640K bytes in size. InitExecSwap's second parameter, SwapFileName, specifies the name and location of the swap file. If EMS memory is available, this name won't be used, but otherwise InitExecSwap will create a new file. InitExecSwap assures that sufficient EMS or disk space exists for the swap, otherwise it returns FALSE. It's a good idea, of course, to put the swap file on the fastest drive that will hold it, to minimize swap times. It's also prudent to avoid a floppy drive, since the user may change disks while the child process is active. The swap file remains open, using a file handle, until ShutdownExecSwap is called or the program ends. InitExecSwap marks the file with the Hidden and System attributes so that the user of the child process won't be tempted to delete it. ExecWithSwap is analogous to the standard Exec procedure in Turbo's Dos unit. Its first parameter is the pathname of the program to execute, and the second is the command line to pass to it. The only difference from Exec is that ExecWithSwap is a function, returning the status of the call in a Word. The function returns DOS error codes, with one exception. Figure 6 lists the most common codes. Figure 6: ExecWithSwap Error Codes 0 Success 1 Swap error (no swap storage, disk error, EMS error) 2 File not found 3 Path not found 8 Insufficient memory You may never need to call ShutdownExecSwap, since ExecSwap sets up an exit handler that automatically calls it when the program ends. In some cases, however, you may want to close and erase the swap file or regain EMS space before continuing. There's a small conundrum here. We've said ExecSwap should be last in the USES list, and we also want the main program to do as little as possible. So where do we place calls to the ExecSwap routines? It's easiest to call them from the main program, and take the hit in overhead. Turbo Pascal provides a better key to the puzzle, though. Version 5 supports procedure variables, and version 4 makes it easy to fake them. So what we do is this: in the main program, assign the address of each ExecSwap procedure to a procedure variable declared in a unit used early in the USES list. Then call ExecSwap's routines in any later unit by referring to the procedure variables. One caution about using ExecSwap: since most of your program's code isn't in memory while the child process runs, it's essential that the program's interrupt handlers be deactivated first. Turbo Pascal 5 provides a handy procedure called SwapVectors that does this for all the System interrupt handlers. Call SwapVectors just before and after ExecWithSwap, and treat any of your own handlers in a similar fashion. Listing 1 offers a simple example of using ExecSwap. You can assemble EXECSWAP.ASM (Listing 3) using MASM 4.0 or later, or any compatible assembler. Then compile the test program to an EXE file and run it, and you'll enter a DOS shell. If you have a DOS memory mapping utility, you'll see that the TEST program is using less than 3K of memory. The swap file uses about 20K, most of that for the 16K stack which is Turbo's default. If the swap goes to EMS, the EMS block will be 32K bytes, since EMS is allocated in 16K chunks. Type Exit to leave the shell and the test program will regain control. A real program provides more impressive results. We developed ExecSwap for use in our Turbo Analyst product, which offers an integrated environment where the programmer can edit source files, then Exec the compiler, debugger, or any of many other programming utilities. Without benefit of ExecSwap, the environment keeps about 250K of memory during the Exec. With ExecSwap, the overhead is only about 4K. That 246K makes a huge difference! How It's Done ------------- ExecSwap's Pascal source file, EXECSWAP.PAS, is given in Listing 2. It's little more than a shell for the assembly language routines in EXECSWAP.ASM, Listing 3. Looking at InitExecSwap in Listing 2, you'll see that it checks first for EMS memory (any version of EMS will do). If that is available, it is used in preference to disk storage. If not, InitExecSwap goes on to assure that there's enough space on the specified drive to hold the swap area. In our production version of ExecSwap (trimmed here for the sake of brevity), we check that the drive doesn't hold removable media. InitExecSwap also stores several items in global variables where they're easily accessible by the assembly language routines, and installs an exit handler to clean up after itself in case the program halts unexpectedly. The tricky stuff is in EXECSWAP.ASM. The file starts with the standard boilerplate needed for linking to Turbo Pascal. We declare a number of temporary variables in the code segment; these are essential because the entire data segment is gone during critical portions of ExecWithSwap. One of these variables is a temporary stack. It's a small one, only 128 bytes, but it is required since the normal Turbo Pascal stack is also swapped out. Macro definitions follow; we've used more than our usual number of macros to keep the listing to a reasonable length. ExecWithSwap starts by copying a number of variables into the code segment. Then it checks to see whether swapping will go to EMS or disk. If neither has been activated, ExecWithSwap exits immediately, returning error code 1. Otherwise, ExecWithSwap processes one of four similar loops: one each to swap to or from disk or EMS storage. Let's trace the "swap to EMS" loop in detail, at label WriteE. The sequence for swapping to disk is so similar that we won't need to describe it here. We first map EMS memory, making the first 16K page of the EMS swap area accessible through the page window at FrameSeg:0. (Note that ExecSwap doesn't save the EMS context; if your application uses EMS for other storage, be sure to remap EMS after returning from ExecWithSwap.) The macro SetSwapCount then computes how many bytes to copy into the first page, returning a full 16K bytes unless it's also the last page. The first location to save is at label FirstToSave, which immediately follows the ExecWithSwap routine. The MoveFast macro copies the first swap block into the EMS window. BX is then incremented to select the next logical EMS page, and the DS register is adjusted to point to the next swap block, 16K bytes higher in memory. The loop continues until all the bytes have been copied to EMS. Next we must modify the DOS memory allocation, so that the space just swapped out is available to the child process. First we save the current allocated size so we can restore it later. Then we switch to the small temporary stack which is safely nestled in the code segment, and finally call the DOS SetBlock function to shrink our memory to just beyond the end of the ExecWithSwap routine. The actual DOS Exec call follows. The implementation here is similar to the one in Borland's Dos unit. It validates and formats the program path and command line, parses FCB's (file control blocks) from the command line in case the child expects them, and calls the DOS Exec function. The error code returned by Exec is stored until the reverse swap is complete. The reverse swap is just that: it reallocates memory from DOS and copies the parent program back into place. There is one critical difference from the first swap, however. Errors that occur during the reverse swap are fatal. Since the program to return to no longer exists, our only recourse is to halt. The most likely reason for such an error is the inability to reallocate the initial memory block. This occurs whenever the Exec call (or the user) has installed a memory resident program while in the shell. Be sure to warn your users not to do this! ExecSwap could write an error message before halting; to save space here, we've just set the ErrorLevel, which can be checked within a batch file: 0FFh can't reallocate memory 0FEh disk error 0FDh EMS error ExecWithSwap is done after it switches back to the original stack, restores the DS register, and returns the status code. The remainder of EXECSWAP.ASM is a collection of small utility routines, some of which may find general use in your library. In Summary ---------- ExecSwap seems quite reliable. It doesn't depend on any newly discovered undocumented features of DOS, and has been tested by thousands of our products' users. There are a few additional features it might have. Our production version writes status messages while swapping, so nervous users don't think their hard disks are being formatted. It might also support direct swapping to extended memory -- we haven't done so because experience indicates that using extended memory in a DOS application is a compatibility nightmare, and RAM disks seem quite adequate for swapping. If the remainder of ExecSwap were converted to assembly language, Turbo Pascal's link order conventions (within a unit) could be circumvented and another 500 bytes or so of Exec overhead would be saved. With a few more DOS memory management calls, it would be possible for the parent and child processes to share a common data area. Finally, an extension of the ExecSwap concept allows TSR programs to leave just a core of interrupt handlers in memory, and swap the application code in when they pop up (SideKick Plus apparently does this). The ExecSwap unit has become a very useful item in our bag of tricks. With an ExecSwap-based DOS shell in the programming editor we use, we can achieve the kind of multitasking we need ("interruption-based" multitasking). ExecSwap should make it easier for you to squeeze more functionality into that 640K box as well. Acknowledgement --------------- Special thanks to Chris Franzen of West Germany, who added disk swapping capability to our original unit, which supported only EMS. This DOC file is an unedited version of an article that appeared in the April 1988 issue of Dr. Dobbs Journal. About the Author ---------------- Kim Kokkonen is the president of TurboPower Software, and the author of many public domain Turbo Pascal tools. He can be reached at P.O. Box 66747, Scotts Valley, CA 95066. Listing 1: TEST.PAS Listing 2: EXECSWAP.PAS Listing 3: EXECSWAP.ASM